Large Language Models as a tool for generating high-level features for text documents

This bachelor thesis investigates the usability of large language models (LLMs) for feature generation from text, evaluating whether LLMs can produce interpretable and usable features for machine-learning tasks. The study uses two labeled datasets: the CORD-19 corpus, consisting of coronavirus research articles with binary labels for high and low citation count, and a dataset of scientific articles from Czech research institutions, with article scores assigned according to the M17+ methodology (ranging from 1 to 5). Seven categorical features were generated for each dataset using the LLama2 language model. These features were used to train models for binary and ordinal classification tasks. Performance was compared to baseline naive models and models trained on term frequency-inverse document frequency (TF-IDF) and sentence embeddings. In the CORD-19 dataset, models using LLM-generated features achieved an accuracy of 59.8%, outperforming the baseline dummy classifier (50.2%) but falling short of TF-IDF (62.5%) and sentence embeddings (62.5%). Combining LLM-generated features with article abstract and title texts using the AutoGluon platform achieved the highest accuracy (66.5%), followed by combining TF-IDF terms and LLM-generated features (65.3%). For the M17+ dataset, the model using LLM-generated features attained an accuracy of 37%, surpassing the naive classifier (18%) and TF-IDF (34.3%). Sentence embeddings achieved the highest accuracy (40.8%), while the AutoGluon model trained on abstract and title text achieved 39.5%. LLM-generated features enhanced the predictive performance of models and demonstrated higher interpretability compared to traditional bibliometric features. However, a notable limitation is the computational cost; generating features for small datasets (2000-3000 samples) requires tens of hours on high-end hardware.