SOURCE-LINKED INTELLIGENCE

Effective Learning Rate Governs Loss Dynamics in Language Model Pretraining

arXiv · AI, language, vision and robotics · article · Aug 25, 2026 · UTC

We uncover ELR collapse in language model pretraining: learning rate (LR) and parameter norm govern loss dynamics primarily through their ratio, the effective learning rate (ELR). When ELR is matched across runs, their loss trajectories collapse throughout training despite substantially different LRs and parameter norms. Across optimizers, architectures, datasets, and model scales, mean collapse errors are typically a few x 10^-3, below the seed-to-seed variation measured in a representative configuration. Systematic ablations identify normalization design and the timescale of LR-norm variatio

Read original source ↗ Open in workspace

recordType

paper

region

Global