Evaluation techniques for the effectiveness of LLM (Large Language Models)

The thesis presents a comprehensive methodology for the systematic evaluation of large language models (LLMs) in the context of an enterprise deployment at Easy Software. The main goal of the thesis is to develop a systematic evaluation framework that allows for an objective assessment of the quality of the prompts before their deployment in a production software environment. It combines theoretical knowledge in the field of language model evaluation with the practical implementation of an evaluation system based on the LangSmith tool and specialized LLM evaluators. The thesis proposes and implements an evaluation methodology that enables systematic comparison of prompts and models based on defined evaluation criteria. At the same time, their validation is performed. The results of the work provide a methodological basis for the evaluations of AI outputs and models in the commercial sphere and can serve as a reference framework for the implementation of similar evaluation processes in other software organizations. The work contributes to a better understanding of the performance criteria of language models and their practical applicability in integrating AI into business applications.