Analyzing Data Lakehouse: The Latest Evolution of Big Data Architectures and Its Benefits for Data Science

The data lakehouse is a new-generation distributed data analytics platform architecture that combines key benefits of data lakes and warehouses into a unified design, reconciling diverse data science and business intelligence workloads over a common data foundation. This thesis examines the novel data lakehouse architecture and a set of associated data preprocessing and data science technologies and demonstrates how data science projects can benefit from them. First, the author describes and analyzes the emerging lakehouse architectural pattern with a primary focus on data storage, integration, and preprocessing as an integral lifecycle stage for data and knowledge mining projects and methods such as machine learning. Second, the thesis describes modules of the Apache Spark ecosystem relevant for machine learning and streaming. Third, the author formulates a subset of challenges observed in current enterprise big data preprocessing and machine learning practice and verifies how integrating described technologies can lead to partially or fully answering such challenges and ultimately increase the business value gained from data science. This work presents a proof of concept implementation illustrating streaming data preprocessing, interactive machine learning feature engineering and modeling, and streaming predictions in the context of a simulated data science project, demonstrating the resolution of the defined data engineering challenges observed in the data science lifecycle, such as supporting data reproducibility and handling upserts, using distributed data lakehouse technologies in a cloud setting.