Empirické porovnání velkých jazykových modelů (LLM)

The aim of this paper is to find criteria suitable for evaluating and comparing large language models (LLM) and leveraging those criteria for model evaluation. The work includes a thorough description of large language models and the principles which they are based on, including an introduction to recent important breakthroughs responsible for their growing popularity. Furthermore, this work focuses on extensive analysis of the state of research and literature in the area of evaluation and comparison of large language models in the domain of software engineering. The aim of the thesis is to develop a methodology for the evaluation and comparison of large language models and to apply this methodology on the most popular large language models. The methodology takes the form of a benchmark and comes with a custom CLI application which automates the comparison of large language models in their ability to develop applications written in the Golang programming language using the Test Driven Development methodology. The models are graded based on their ability to fulfill functional requirements and iteratively develop applications which pass a set of automated tests. The generated applications are then reviewed in the qualitative analysis section, where code quality and adherence to best software engineering practices is checked. Also included is an analysis of the strengths and weaknesses of all selected models and recommendations are provided on what each model is suitable for and where caution is advised.