Využití a porovnání architektur umělých neuronových sítí pro detekci anomálií v síťovém provozu

This thesis explores the use and comparison of artificial neural network architectures for anomaly detection in network traffic. With the increasing sophistication of cyber threats and the limitations of traditional signature-based approaches, machine learning and deep learning algorithms are becoming a promising tool for identifying previously unknown attacks. The goal of this thesis is to conduct a review of current anomaly detection methods with a focus on artificial neural network architectures, design an experimental framework, and systematically compare the detection capabilities, operational efficiency, explainability, and transferability of selected architectures. A total of eight architectures were implemented, covering classical machine learning (LightGBM, SVM), sequential deep learning (Bi-LSTM), the attention mechanism (TabTransformer), and reconstruction-based approaches (autoencoders, SAE+SVM, MemAE+EIF, Kitsune). The experiments were conducted on four public benchmark datasets (CICIDS2017, CSE-CIC-IDS2018, NSL-KDD, UNSW-NB15), a specialized IoT dataset (IoT-23), and two custom datasets collected from a real enterprise network. The results show that the tree-based LightGBM algorithm represents the most versatile solution with a favorable trade-off between detection performance and computational cost, while attention-based architectures are primarily justified for data with complex categorical features. Reconstruction-based models proved to be a viable alternative for IoT environments without labeled data, albeit at the cost of a higher false alarm rate. Transfer learning experiments on custom unannotated data demonstrated that direct deployment of models into a new network environment without domain adaptation is highly unreliable. Explainability analysis using the SHAP framework revealed fundamental differences in the learning strategies of the tested architectures and confirmed the critical role of explainable artificial intelligence methods in detecting hidden data bias.