SOURCE-LINKED INTELLIGENCE

Terminal Shrinkage Averaging Reveals a Schedule-Estimator Interaction in LLM Pretraining

arXiv · AI, language, vision and robotics · article · Sep 21, 2026 · UTC

Large language model (LLM) pretraining conventionally returns the raw final iterate. This couples two design choices: the learning-rate schedule that generates the parameter trajectory and the estimator that constructs the deployed model (e.g. the raw final iterate or a checkpoint average). A schedule that promotes optimization progress may differ from one that minimizes variation in the raw final iterate. Separating these choices creates an opportunity to maintain progress late in training while reducing variation in the returned model. To this end, we propose \emph{Terminal Shrinkage Averagi

Read original source ↗ Open in workspace

recordType

paper

region

Global