Technique
Distillation
Training a small 'student' model to mimic a large 'teacher' model's outputs.
Definition
Distillation transfers knowledge from a big model to a small one by training the student on the teacher's logits, generated tokens, or hidden states. The student inherits much of the teacher's quality at a fraction of the inference cost. Phi, Gemma and many edge models were distilled.
Common use cases
- Edge models
- Cost reduction
- Latency optimisation