SOURCE-LINKED INTELLIGENCE

iSDFT: Information-Proximal Self-Distillation for Continual Learning in LLMs

arXiv · AI, language, vision and robotics · article · Sep 21, 2026 · UTC

On-policy self-distillation fine-tuning (SDFT) learns new skills from demonstrations while reducing forgetting, but it always distils toward the full demonstration-conditioned teacher. This fixes teacher influence at the full-teacher endpoint, providing no control over how much demonstration information should be transferred at each prediction state. We introduce Information-Proximal SDFT (iSDFT), which instead treats the teacher as a budgeted source of information. At each token, iSDFT selects the distribution closest to the current student that satisfies a prescribed teacher-information cons

Read original source ↗ Open in workspace

recordType

paper

region

Global