Distillation
Distillation is like a student learning from a master teacher; instead of reading every textbook from scratch, the student watches the teacher's final answers to learn how to solve problems quickly and efficiently.
Distillation is a model compression technique where a compact 'student' model is trained to minimize the divergence between its output distribution and that of a pre-trained, larger 'teacher' model, effectively distilling the teacher's generalized knowledge into a more computationally efficient architecture.
Training a smaller 'student' model to mimic the outputs of a larger 'teacher', transferring capability into a cheaper, faster model.
evolution
- 2006 · historyBucilă et al. Model Compression
Researchers introduced the concept of compressing a large ensemble of models into a single, smaller neural network.
- 2015 · historyHinton et al. Knowledge Distillation
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean formalized 'Knowledge Distillation' as a method to transfer knowledge from a large teacher model to a smaller student model.
- 2019 · historyDistilBERT
The release of DistilBERT demonstrated that distillation could effectively reduce the size of large-scale Transformer models like BERT while retaining most of their performance.
- 2023 · historyInstruction-tuned Distillation
Models like Alpaca popularized using large proprietary models to generate synthetic training data to distill reasoning capabilities into smaller open-source models.