SOURCE-LINKED INTELLIGENCE

Does On-Policy Distillation Really Distill? From Noisy Teacher to Self-Improvement

arXiv · AI, language, vision and robotics · article · Aug 31, 2026 · UTC

On-policy distillation (OPD) offers dense token-level supervision as an alternative to the sparse outcome-level advantages of reinforcement learning with verifiable rewards (RLVR). However, the teacher scores student-generated trajectories that are inherently off-policy for it, so the reliability of its supervision, and hence the source of the student's improvement, remains unclear. We quantitatively analyze teacher supervision during OPD training and find substantial noise whose prevalence increases with teacher scale. Surprisingly, the student policy is insensitive to such noise, converging

Read original source ↗ Open in workspace

recordType

paper

region

Global