Automated data correction in near real-time processing (NRT)

The bachelor thesis deals with the improvement of data quality in near real-time (NRT) processing. In the theoretical part of the thesis, based on a research, it defines the concept of NRT (latency in the order of milliseconds to lower units of seconds) and identifies domains where NRT processing brings added value. Furthermore, data quality is characterized using seven dimensions: accuracy, timeliness, completeness, uniqueness, relevance, validity (syntactic correctness) and consistency. At the same time, key challenges that complicate the fulfilment of these dimensions in an NRT environment are described. In the practical part, an NRT pipeline built on Kafka and Flink 1.20 (PyFlink) tools was designed and implemented as the basis for two prototypes of automated heat pump data correction. The first prototype was shown to increase data accuracy and completeness and to meet the average end-to-end delay limit set when defining the NRT concept. The second, experimental prototype, based on inaccuracy modelling using additional heat pump sensors, confirmed that even a more complex model can be operated within the required limits, albeit with a lower impact on accuracy. The results demonstrate that a well-designed architecture and targeted correction mechanisms can significantly improve data quality in NRT processing without violating the delay requirements. The work also identifies directions for further development, in particular streamlining processing within Flink Job, more advanced preprocessing, and a possible transition to a Java-based Flink implementation to achieve latencies on the order of tens of milliseconds.