Large Language Models (LLMs): Examining the quality of generated text with task specific data

Large Language Models (LLMs) have emerged as powerful tools for generating human-like text across various applications. This thesis examines the factors influencing the success of LLMs in text generation tasks, focusing on prompting techniques and the integration of external data within the prompt. The research aims to validate the role of prompt engineering in generating compelling content, particularly in the context of writing marketing newsletters. The thesis begins by introducing the research area of LLMs and prompting, followed by a literary review of relevant prompt engineering techniques. Based on the findings, a set of hypotheses is formulated, outlining the potential impact of these techniques on text generation quality. A controlled experiment is conducted to validate the hypotheses, applying the identified prompting techniques to the task of writing email newsletters with a single LLM (GPT-3.5-turbo-1106). The generated newsletters are evaluated using quantitative metrics and A/B/n test campaign to an audience (n=550) of registered attendees of a university event. The results of the experiment confirmed the important role of prompting techniques and external data. On the specific task of newsletter writing, usage of advanced prompting techniques resulted in 15% improvement in click-through rate for a given prompting technique leveraging external data (Improving the click-through rate from 4% to 19%), demonstrating prompt engineering and external data play an important role when creating effective texts that engage the audience. Techniques leveraging specification of tone of voice, providing additional context data, relevant examples or industry best practice instructions have shown to have the largest impact on the resulting text and achieved superior performance on the measured task compared to a base prompt solution without prompting techniques.