An Empirical Study on the Evaluation of Large Language Models in the Czech Language and the Validation of "LLM-as-a-Judge"

This thesis focuses on the evaluation of large language models (LLMs) in the Czech language context and examines the validity of the LLM-as-a-Judge approach as a method for automated assessment of model outputs. The motivation for this work stems from the limited representation of Czech in existing evaluation studies and the increasing importance of reliable evaluation of generative models in practical applications. The main objective is to assess the performance of selected language models on tasks related to the Czech language and mathematics, and to evaluate the level of agreement between human evaluation and automated evaluation performed by another language model. An experimental methodology is proposed, involving the testing of different types of models, including proprietary, open-source, and locally deployed solutions. The models are evaluated based on predefined criteria, and the results are analysed with respect to response quality, differences across task types, and output consistency under repeated generation. Special attention is given to the impact of generation parameters on the stability of model outputs. The results indicate significant differences in performance across models and show that the LLM-as-a-Judge approach achieves a relatively high level of agreement with human evaluation, although certain systematic deviations are observed. The thesis contributes to a better understanding of the capabilities and limitations of LLM evaluation in the Czech language and provides recommendations for their practical use.