SOURCE-LINKED INTELLIGENCE

TV-Regulated OPD: Direction Matters in On-Policy Distillation

arXiv · AI, language, vision and robotics · article · Sep 8, 2026 · UTC

On-Policy Distillation (OPD) facilitates the transfer of knowledge from domain expert to student in the post-training phase of Large Language Models (LLMs). However, the supervision signals in mainstream OPD methods suffer from high variance and noise which is generally instable during training. In this work, we systematically investigated what really matters to the performance and the fundamental mechanisms behind the instability during training. We found that retaining only the sign of token-level advantages is sufficient to achieve the performance comparable to standard OPD. Meanwhile, smoo

Read original source ↗ Open in workspace

recordType

paper

region

Global