Řízení datové kvality v Hadoop Ecosystem

This diploma thesis focusses on data quality management in Hadoop Ecosystem tools. The aim of this thesis is to create a prototype tool for data validation and monitoring in the Threat Intelligence department of a cyber security company. The thesis defines the concept of Big Data, its relationship and significance for a data-driven company and specific Big Data technology Hadoop. Hadoop is described including Hadoop Ecosystem tools and its architecture. Furthermore, the thesis focuses on a data-driven company, its processes, human resource management and technical prerequisites. Thesis then focuses on a specific data department of the company, identifies its processes, data sources and the problem with reporting. This problem is solved through the development and subsequent implementation of a tool for data validation and monitoring. The tool is programmed in Python using PySpark, implemented in the company's infrastructure and integrated into the Slack. The conclusion evaluates the effects of the validation tool and the possibilities for future development.