Term
Knowledge Distillation
Knowledge distillation is a training technique in which a small student model learns from the behaviour of a large teacher model — aiming for comparable quality at significantly lower resource cost.
Knowledge distillation — explained in more detail
Introduced in 2015 by Hinton et al., distillation is the standard way to compress a powerful but expensive model into a smaller, cheaper variant. The teacher model produces predictions or soft probabilities over a dataset; the student is trained to reproduce these outputs as accurately as possible — typically with a mix of cross-entropy against the true labels and KL divergence against the teacher distribution. In the LLM era the term has broadened: often the student is simply trained on answers that the teacher has generated for a prompt catalogue — a form of synthetic training data. The benefit: many properties of the large model (style, reasoning steps, factual knowledge) transfer over without the student ever having seen original data.
Example / Practical context
Well-known examples: DistilBERT (student of BERT, ~40% smaller, ~60% faster, ~97% of the quality), Alpaca (LLaMA-7B student trained on 52k responses from GPT-3.5), and many Llama and Mistral forks distilled on GPT-4-generated data. Practical value: a local 7B model can come surprisingly close to a 70B or closed-source model on narrow tasks — at a fraction of the inference cost.
Distinction from related concepts
Quantisation shrinks a model through lower bit-width weights — without teaching new behaviour. Pruning removes unimportant weights. Both modify the same model, whereas distillation trains a new, smaller model. Classic SFT on teacher answers is today the most common practical form of distillation; some provider licences explicitly forbid this use.
Entdecke mehr
AI Workflows by Keyword: How We Make Recurring Routines Enforceable
A typed keyword triggers a fixed AI routine — and every single step must be committed before the next one appears. Why that's the actual trick.
GlossarDPO (Direct Preference Optimization)
DPO is a fine-tuning method that aligns language models directly to human preference pairs — without a separate reward model and without reinforcement learning.
LexikonFine-Tuning Explained — When, How, With What
When fine-tuning pays off, which methods exist (SFT, DPO, LoRA, QLoRA), and what AMD vs. NVIDIA hardware actually means in practice.