On-Policy Distillation: 30 times cheaper to train your own reasoning model
Thinking Machines Lab — the research lab founded by Mira Murati — published a detailed report on On-Policy Distillation in October 2025. The method has since been adopted by Qwen3, MiMo, and GLM-5 in their post-training pipelines, and follow-up arXiv papers released in April 2026 examine the phenomenology in more detail. The practical finding: reasoning capabilities can now be transferred to smaller models at a fraction of the cost.
What the method delivers in concrete numbers
- 70 percent on AIME’24 with Qwen3-8B as the student model — comparable to far larger pure-RL training runs.
- 30× cost reduction compared with classical off-policy distillation on identical hardware.
- No reward model needed: instead of sparse outcome rewards, the teacher provides token-by-token log-probs as a dense training signal.
- Industry validation: Qwen3, MiMo, and GLM-5 use the technique productively in their post-training stages.
What was true before
Anyone wanting to lift a smaller model to the reasoning level of a larger one had two options. Off-policy distillation: the large model produces a dataset of example answers, the small model is trained on it via supervised learning. Fast, but the student only sees states the teacher visits — not its own. As soon as the student deviates from the teacher trajectory in practice, it lacks the correct next steps. This phenomenon is called exposure bias and has been a known problem for years.
Full reinforcement learning: the student generates its own answers and is trained via reward signal (e.g. “right/wrong” on math problems). Solves the exposure-bias problem but is expensive — the reward arrives only once per answer, the training signal is sparse, convergence is slow.
What is true now
1. Student generates, teacher gives token-by-token feedback. With on-policy distillation, the student model actually runs and produces its own rollouts. Rather than receiving a binary reward at the end, the method compares every generated token with the probability distribution the teacher model would have had for exactly that context. The training signal becomes dramatically denser — per token rather than per answer.
2. The math behind it is simple. The training loss is the Kullback-Leibler divergence between student and teacher distribution at every position of the student rollouts. No reward model, no PPO clipping, no reference model — only a teacher correcting the student along its own trajectory. Thinking Machines Lab themselves call it “cheap accuracy, real gains”.
3. Concrete numbers. With Qwen3-8B as student and a stronger model as teacher, the method reaches 70 percent on AIME’24 — a math benchmark that has served as the litmus test for serious reasoning models. The training compute needed to reach this level is about 1/30 of what classical off-policy distillation costs.
Why it matters
The method does not solve an open theoretical problem — on-policy training signals have been known in the RL literature since the 1990s. What makes it relevant is its practical reach: three of the most relevant Chinese model families (Qwen3, MiMo, GLM-5) have adopted it in their post-training pipelines according to the follow-up analysis, and the corresponding Tinker Cookbook code from Thinking Machines is publicly available as a recipe.
The consequence for the coming months is foreseeable: the threshold above which a lab or small company can sensibly train its own reasoning models is dropping visibly. Two years ago that required a cluster and six-figure compute budgets. Today a good teacher model and roughly one-thirtieth of the compute needed for pure RL is enough. The effect is already visible in the speed at which open-weights reasoning models have appeared on Hugging Face over the last six months.
Important for the reality check: the method does not replace large-scale pretraining, nor RL for entirely new capabilities. It transfers existing capabilities efficiently — and that is what most productive use cases actually need.
What you can do now
If you want to fine-tune a model for a domain yourself: look at on-policy distillation as an alternative to classical SFT. The Tinker Cookbook distillation recipe (github.com/thinking-machines-lab/tinker-cookbook) is a good entry point.
If you need a teacher: stronger open-weights models like Qwen3 or Llama reasoning variants are freely usable. Closed-source models as teachers usually fail because the necessary token log-probs are not exposed via API.
If you are only a consumer: expect small specialist models (3B–8B parameters) to reach reasoning levels in the coming months that were only possible for 70B models a year ago. That visibly shifts the economics of edge and on-premise deployments.
Entdecke mehr
AI Workflows by Keyword: How We Make Recurring Routines Enforceable
A typed keyword triggers a fixed AI routine — and every single step must be committed before the next one appears. Why that's the actual trick.
GlossarDPO (Direct Preference Optimization)
DPO is a fine-tuning method that aligns language models directly to human preference pairs — without a separate reward model and without reinforcement learning.
LexikonFine-Tuning Explained — When, How, With What
When fine-tuning pays off, which methods exist (SFT, DPO, LoRA, QLoRA), and what AMD vs. NVIDIA hardware actually means in practice.