Implementation of near-real time big data application in a real-world setting

This thesis focuses on the design and implementation of a near-realtime big data application in the Data Lake concept according to the proven methodologies of Komerční banka. The main goal is to create a new application named LAKE DEUS, which will enable efficient processing and distribution of large volumes of data in near-real time, while ensuring high quality, availability, and reliability of data. This work delves deeply into big data concepts and methods of their real-time processing. A key element is a detailed analysis of the architecture of the new LAKE DEUS application, designed to implement these concepts. The text further elaborately discusses the phases of creation and deployment of the application, its streaming connection to the data source, subsequent processing, and making data available to subscribers. It focuses on advanced methods of data processing and modeling, process automation, and data quality monitoring. One of the key points of the work also consists of comparing and evaluating various methods and technologies used in the application's implementation. Based on these analyses, recommendations for the future development of the project are formulated. The result of this work is a comprehensive near-realtime big data application that meets the methodological requirements of Komerční banka and will serve as a model for future projects in the field of big data and near-realtime data processing on a big data platform in the Data Lake concept.