ETL process optimization

At a certain point of its life cycle, each data warehouse gets into a state in which the length of the ETL processing starts slowly approaching the acceptable limit. Even the slightest lapse means a hardly erasable time lag and the data supplies to users are increasingly delayed. Before the situation becomes more problematic, we need to react. This diploma thesis deals with the possible solutions of this problem- the ETL performance optimization. The main aim of the thesis is to propose the set of recommendations for the ETL process optimization and its presentation based on practical real world examples. While going through this thesis, the readers get inside the running of the real ETL optimization project. They learn how to proceed during its particular periods, what to avoid and how to reduce the risk level as much as possible. Moreover, the thesis mentions the prevention of the ETL performance loss and discusses the critical factors of the effective ETL process design. Although the proposed methods are described in a general perspective, the thesis is mainly focused on Oracle's database, which is its certain limitation. Also hardware optimization is not a part of the text. First chapters of the thesis describe the problem from the theoretical point of view, which is important for the correct understanding of the main content and the existing literature concerning this problematic is included here as well. Next sections describe the proposed methods and recommendations for the optimization of the ETL processes including tests and project results evaluation. They are mainly focused on the ETL process synchronization and the SQL code optimization areas. All the methods are presented of examples of the real project implemented by the author.