AIIC AI Intelligence Centre

SOURCE-LINKED INTELLIGENCE

Less Loss, Fixed Weights: Logit Corrections for Forgetting

arXiv · AI, language, vision and robotics · article · Oct 30, 2026 · UTC

When a language model is fine-tuned to follow instructions, or further pretrained on a new subject, it often gets worse at predicting standard web text. This is known as "forgetting." We ask whether this loss can be undone by simple, fixed adjustments to the model's final token predictions. Across 14 pairs of base and derived models, from 1B to 7B parameters, we fitted a fixed bonus or penalty for each token (a vocabulary bias), alone or together with a temperature. On text held out from correction fitting, the largest top-4K bias recovery estimate is under 10%, with loss conditional on the re

Read original source ↗ Open in workspace

recordType
paper
region
Global

Evidence & attribution

First collected: 2026-11-02T10:21:21.339Z. This is not the publication date.