University of Economics log patternisation

This diploma thesis focuses on the design and implementation of a system for automated processing and analysis of logs at the University of Economics in Prague. The work addresses the problem of insufficient cybersecurity monitoring, where only approximately 30 out of more than 2,200 log sources have been analyzed so far. The implemented solution uses the Extended Nagappan-Vouk (ENV) algorithm for effective identification of patterns in log data, achieving an F1 score of 96.0 %. For performance optimization, parallel processing techniques and Apache Parquet columnar storage were implemented, reducing file sizes by 70-85 % and accelerating data loading by up to 90 %. The system shortened the processing time of monthly logs from the original 12 hours to just 15 minutes, representing a 48-fold speed improvement. An interactive dashboard developed using the Streamlit library forms part of the solution, providing a user-friendly interface for visualization and interpretation of results. The designed system significantly strengthens the university's cybersecurity by improving the detection of potential security incidents and enables efficient filtering and analysis of logs. The thesis also discusses possibilities for further development, including the implementation of real-time processing and the use of machine learning techniques for automatic anomaly detection.