What is model distillation for LLMs?

Model distillation (knowledge distillation) for LLMs is a training technique in which a student model learns to imitate useful behavior from a more capable teacher model. The student is often smaller, making it potentially cheaper and faster to deploy.

The goal is not to copy the teacher's parameters. It is to transfer some of the teacher's learned behavior into a separately trained model.

Simple example

Imagine a company uses a large LLM to classify customer-support tickets into billing, technical, or account issues.

• The teacher LLM generates high-quality classifications and explanations for representative tickets.
• A smaller student is trained on those examples.
• After evaluation, the student may handle this narrower task without calling the expensive teacher for every request.

This can reduce serving cost and latency if the student meets the required quality standard.

What does the student learn from?

Different distillation methods use different teacher signals:

• Generated answers or labels: The teacher produces example responses that become supervised training targets.
• Soft token probabilities (logits): When available, the student learns from the teacher's distribution over possible next tokens, not only the chosen token.
• Explanations or rationales: Carefully generated intermediate explanations can provide additional training signals for some tasks.

The teacher and student need not share the same architecture. However, token-probability distillation can require additional alignment work when their vocabularies or tokenizers differ.

Why distill an LLM?

• Smaller deployed model: Lower parameter count can reduce weight-memory requirements.
• Potentially faster responses: A smaller student may improve latency or throughput on suitable hardware.
• Lower serving cost: Repeated requests may become cheaper once the student is trained.
• Task specialization: The student can learn strong behavior for a narrower, well-defined workload.

Distillation vs. quantization vs. fine-tuning

• Distillation: Trains a *student model* using signals from a teacher.
• Quantization: Stores or computes using lower numerical precision; it does not inherently train a new student.
• Ordinary supervised fine-tuning: Updates a model using labeled examples. Distillation may use fine-tuning internally, but the key difference is that the teacher supplies the learning signal.

These approaches can be combined; for example, a distilled student may later be quantized.

Does the student become as good as the teacher?

Not automatically. A student can inherit teacher mistakes, lose abilities outside its training distribution, and struggle on tasks that exceed its capacity. Preparing teacher-generated training data also takes compute, time, and appropriate usage rights.

Evaluate the student on held-out, representative tasks and measure quality, latency, memory, and cost before deployment.

For the training procedure in more detail, see “How does LLM distillation work?”

In short: LLM distillation trains a usually smaller student to reproduce selected behaviors of a stronger teacher, trading training effort for a potentially more efficient model.

References: Hugging Face: Distillation Trainer, Distilling Step-by-Step!, and Distilling the Knowledge in a Neural Network.