Exploration of model distillation technique for efficient deep learning of classification transformers

This thesis explores knowledge distillation as a technique for efficient deep learning of classification tasks with transformer based models. With the increasing size and computational demands of modern neural networks, knowledge distillation is becoming an efficient tool for transferring capabilities of larger teacher models into smaller and more efficient students. This work experimentally evaluates the benefits of knowledge distillation and investigates the conditions under which it yields the best results. A series of experiments was designed and conducted using various types of model architectures on both textual and image datasets. The models were compared based on their classification performance, size and inference speed. The results show that knowledge distillation provides the biggest benefit when applied to randomly initialized students, whereas its effect on pre-trained models during fine-tuning is limited. The thesis also demonstrates the synergistic effects of combining distillation with other efficiency focused techniques such as data augmentation and logit precomputation that lead to more efficient model training overall. The experimental results confirm the effectiveness of knowledge distillation while also identifying its limitations and key factors that influence its success.