On-Policy Distillation: Frontier Reasoning on Small Models

What you need to know

• The idea in one line. The small "student" model generates its own answers, and a stronger "teacher" grades those answers token by token — so the student learns from its own mistakes, not from a transcript it can only mimic.
• Why it beats plain fine-tuning. Copying a teacher's perfect outputs (off-policy) makes small errors compound over long reasoning chains. On-policy learning trains the model on the states it actually visits, which is exactly where it needs help.
• Why it beats reinforcement learning on cost. RL gives one sparse reward per trajectory; on-policy distillation gives a dense per-token signal. Thinking Machines Lab reports this is roughly 9 to 30 times cheaper in compute to reach the same score.
• The evidence is stacking up. An April 2026 survey, a mechanism paper, and a 35B agent that matches trillion-parameter models all point the same way — reasoning transfers well into small models when the student stays on-policy.
• The builder payoff. Near-frontier reasoning on modest GPUs is now a realistic in-house project — the practical way for India and UK teams to dodge US API costs and keep data in-region.

For two years the reasoning conversation has been about scale: bigger models, longer chains of thought, more test-time compute. That is expensive to run and, for most teams, expensive to rent from a US provider one metered token at a time. The quieter story of 2026 is a family of techniques — grouped under on-policy distillation — that move frontier-grade reasoning into models small enough to fit on a single node and cheap enough to serve at volume. It is not a new model release; it is a training method, and it changes what a self-hosting team can realistically build. Let us take it apart and then talk about what you should actually do with it.

Off-policy versus on-policy: what actually changed

Classic knowledge distillation is off-policy. The teacher produces a set of answers, you freeze them, and the student is trained by supervised fine-tuning to reproduce those answers. It works, and it is cheap, but it has a structural flaw for reasoning: the student only ever sees the teacher's clean, correct paths. At inference the student inevitably drifts off those paths — it makes a small slip in step three — and because it never trained on that situation, it has no idea how to recover. The errors compound. The longer the reasoning chain, the worse it gets. The April 2026 Survey of On-Policy Distillation for Large Language Models (Song and Zheng) frames this precisely as error accumulation across student-generated sequences, and it is the core reason plain fine-tuning underperforms on hard, multi-step tasks.

The on-policy fix

On-policy distillation flips the data source. The student generates the trajectories — its own attempts, mistakes and all — and the teacher supervises those. Now the training signal lands exactly on the states the student actually visits, including the messy recovery situations that off-policy training never shows it. This is the same insight that makes reinforcement learning work, but with a crucial difference in the reward.

The reverse-KL trick

Reinforcement learning gives you a sparse reward: one score for the whole answer, leaving the model to guess which of hundreds of tokens deserved credit. On-policy distillation replaces that with a dense, per-token signal. At each step, the teacher scores how far the student's next-token distribution has drifted from its own — a per-token reverse Kullback-Leibler divergence, computed with a single teacher forward pass over the student's trajectory. Thinking Machines Lab's widely-cited essay On-Policy Distillation explains why reverse KL is the right choice: it is "mode-seeking", pushing the student to commit to the teacher's actual behaviour at each state rather than smearing probability across plausible-but-wrong options. Every token carries information, so the student learns far faster per rollout than RL allows.

How the three approaches compare

The useful mental model is a spectrum of who generates the data and how densely the teacher supervises it. Off-policy SFT is cheapest but brittle; RL is the most general but sample-hungry; on-policy distillation sits in between and, for reasoning tasks where a good teacher exists, tends to win on cost-for-quality.

Cost multiples are as reported by Thinking Machines Lab for specific reasoning experiments; treat them as direction, not a guarantee for your workload. If you already run a GRPO pipeline, our guide to reinforcement fine-tuning with GRPO is the natural comparison point.

The practical reading: on-policy distillation is not a replacement for RL, it is the cheaper first choice whenever a stronger model already does the task well. You reach for RL when there is genuinely no teacher — when you are trying to push a model past every model that exists. For the overwhelming majority of builder tasks, a teacher exists, so distillation is the pragmatic route.

The evidence: small models, big reasoning

The reason this is worth your attention now, rather than as a curiosity, is that the numbers have arrived. In Thinking Machines Lab's reasoning experiment, a Qwen3-8B student that had plateaued after supervised fine-tuning was pushed from roughly 60% to around 70% on the AIME'24 maths benchmark in about 150 on-policy steps, using a Qwen3-32B teacher. They cite the Qwen3 technical report showing on-policy distillation reaching 74.4 on AIME'24 against 67.6 for reinforcement learning alone. Crucially, they report the whole process costing on the order of 9 to 30 times less compute than the RL route to the same quality — the low end when the fine-tuning data is amortised, the high end when you count the full teacher cost for a fresh task.

The technique also repairs models, not just trains them. In a personalisation experiment, mid-training a model on new domain data wrecked its instruction-following: training on documents alone crashed IF-eval from around 85% to 45%, and even a 70/30 mix of documents to chat data still dragged it down to 79%. A short bout of on-policy self-distillation, using an earlier checkpoint of the same model as the teacher, recovered instruction-following from that 79% back to roughly 83% — without losing the newly-learned domain knowledge. That "learn a new skill without forgetting the old one" property is exactly the pain every team hits when it fine-tunes a small model in production.

The most striking 2026 result scales the idea up. Scaling the Horizon, Not the Parameters introduces Agents-A1, a 35B Mixture-of-Experts agent trained with multi-teacher, domain-routed on-policy distillation — a specialised teacher per domain, all distilled into one student. The paper reports it matching or beating trillion-parameter frontier models such as Kimi K2.6 and DeepSeek V4-Pro on agentic benchmarks (for example BrowseComp at 75.5 and HLE at 47.6). A 35B model that behaves like a trillion-parameter one is the whole promise of this research line in a single result: you scale the quality of the training signal, not the parameter count you have to host.

This connects directly to what you can already self-host. A strong open-weight teacher like NVIDIA's Nemotron 3 Ultra or a frontier Chinese open weight can play teacher on your own hardware, so the entire pipeline — teacher and student — can live in-region with no data leaving your control. For the general production shape of shrinking a model while holding quality, our companion guide on teacher-student distillation in production covers the plumbing.

What this means for India and UK builders

The strategic point is about optionality. Until recently, a team that needed strong reasoning had two options: pay a US frontier API per token, or self-host a giant open weight and eat the GPU bill. On-policy distillation adds a third: take a task where a big model does well, distil that behaviour into a small student, and serve the student cheaply at volume. For narrow, high-throughput jobs — a support triage agent, a code reviewer, a claims classifier, a document extractor — this is frequently the best economics available.

For Indian teams, it maps neatly onto the compute that is now subsidised. IndiaAI-mission GPU-hours are aimed at exactly this kind of training-and-serving workload, and because the student is small and the teacher only does forward passes, a distillation run fits comfortably inside a modest allocation. For UK teams, sovereign-compute and AI Research Resource programmes give the same route without depending solely on the hyperscalers. In both markets the data-residency argument is decisive: if you handle personal data under India's DPDP framework or UK GDPR, keeping both teacher and student in-region sidesteps the whole question of shipping prompts to a foreign API.

How to actually do it

You do not need to implement the maths from scratch — the open-source ecosystem has moved quickly, and several training libraries now ship on-policy distillation trainers. But you should understand the loop, because the failure modes live in it. Conceptually, one training step looks like this:

# Conceptual on-policy distillation step (illustrative, not runnable).
# student  = small model you can self-host and will ship
# teacher  = stronger frozen model (open-weight, self-hosted, or an API)

for batch in prompt_loader:
    # 1. Student samples its OWN answers  (this is the "on-policy" part)
    trajectories = student.generate(batch, num_samples=4)

    # 2. Teacher scores the student's tokens — no waiting for a final reward.
    #    One teacher forward pass gives a dense, per-token signal.
    teacher_logprobs = teacher.forward(trajectories)      # frozen
    student_logprobs = student.forward(trajectories)

    # 3. Per-token reverse KL: pull the student towards the teacher at each state
    per_token_loss = student_logprobs - teacher_logprobs  # reverse KL term
    loss = per_token_loss.mean()

    # 4. Update only the student
    loss.backward()
    optimizer.step()
    optimizer.zero_grad()

Three things make or break this in practice. First, the teacher choice: pick one whose reasoning style your small student can actually represent, and that is genuinely stronger on your task — the mechanism paper is blunt that a badly-matched teacher gives you nothing. Second, the warm-start: run a short round of cheap off-policy SFT first so the student is competent enough to generate usable trajectories, then switch to on-policy for the reasoning polish. Third, the eval loop: score the student on held-out problems every few hundred steps, because on-policy training can quietly drift if the teacher signal and your real objective diverge. If you are still deciding whether to train at all, our fine-tuning decision ladder is the right gut-check before you provision anything.

Once you have a distilled student, it slots into the same serving and agent stack you already run. A small reasoning model is a natural brain for a local agent — the pattern in our walkthrough on building a local AI agent with Ollama, a small model and MCP tools — and it keeps the reasoning on your own hardware end to end.

So — should you use it?

For most builders in India and the UK, on-policy distillation is now the default answer to a specific question: "how do I get reliable reasoning on a task I run at volume, without paying a frontier API forever?" It is not the answer to "how do I build something no model can do yet" — that is still RL and research territory. But those genuinely novel problems are rare. The common case is a task a big model already handles well, that you need cheaper, in-region, and under your control. That is exactly what this technique is for. Write the eval, pick a well-matched teacher, warm-start with a little SFT, then let the student learn from its own attempts under supervision. The research of 2026 has made the case; the economics make it worth acting on. A 35B model that reasons like a trillion-parameter one on agentic tasks is not a headline — it is a budget line you can now put on your own roadmap.