TL;DR: Distillation trains a small "student" model to imitate a large "teacher" model, transferring much of the capability into something far cheaper to run. It is why the fast, inexpensive tier of every model family exists, and why small open models keep punching above their weight.
How it works
Ordinary training teaches a model from raw data. Distillation teaches it from another model: run the big teacher on a large set of prompts, collect its outputs, and train the compact student to reproduce them. The student learns from a much richer signal than the internet ever provides - instead of guessing what a good answer looks like, it studies a strong model's actual answers, including how it structures reasoning and handles edge cases. In the classic formulation the student matches the teacher's full probability distribution over next tokens (seeing that "couch" was nearly as likely as "sofa" carries information a single right answer does not); in the common modern recipe it simply fine-tunes on teacher-generated text.
The economics explain its ubiquity. A frontier model gives the best answers but costs the most per request and responds slowest. For high-volume production work - support triage, extraction, classification, autocomplete - a distilled model at a fraction of the size often delivers nearly the same quality on that narrower distribution of tasks at a fraction of the price and latency. Model families are built this way on purpose: the small fast tier is trained with heavy help from the large one, which is a key input when you are choosing a model tier. Reasoning ability distills too: training a student on a reasoning model's worked step-by-step solutions transfers a useful share of that skill into models small enough for local inference.
The limits are real, though. The student compresses the teacher, and what gets lost is breadth: performance holds up best on the kinds of prompts the distillation data covered and sags on long-tail queries the teacher would have handled. A student also inherits its teacher's flaws - biases and confident errors are copied along with the skills. And legally, most hosted providers' terms prohibit using their outputs to train competing models, which is why serious distillation pipelines run on models with permissive licenses or in-house teachers.
Distillation and quantization are complementary, not competing: distillation produces a smaller network, quantization stores whichever network you have in fewer bits. Production small models are routinely both distilled and quantized.
Where it sits in the AI stack
Distillation is a training-time compression step that turns an expensive teacher into a deployable student:
Key tools and implementations
-
Small tiers of model families
The fast, cheap sibling in each major lineup, trained with heavy help from its larger kin.
-
Distilled reasoning models
Students trained on a reasoning teacher's worked solutions, bringing step-by-step skill to small sizes.
-
Synthetic data pipelines
Teacher-generated training sets - the raw material most modern distillation runs on.
-
Hosted distillation services
Provider workflows that tune a smaller model from your logged large-model traffic.
Related entries
- Quantization Storing a model's weights in lower-precision numbers so it needs less memory and runs faster.
- Fine-tuning Further training a pretrained model on your own examples to specialize its behavior for a task.
- Choosing a model A framework for weighing capability, latency, cost, privacy, and openness when picking a model for a task.
- Local LLM Running a language model entirely on hardware you control instead of calling a hosted API.