SOURCE-LINKED INTELLIGENCE

Does Step Law Transfer to Small-Scale Language Models? An Empirical Recalibration Below 59M Parameters

arXiv · AI, language, vision and robotics · article · Sep 23, 2026 · UTC

Step Law gives power-law formulas for the optimal peak learning rate eta* and batch size B* when pre-training language models. It was calibrated on models between 59M and 1B parameters; the small-model regime N < 59M was never tested empirically by its authors. This regime matters for single-GPU training, interpretability research, educational experiments, and settings where larger models are infeasible on memory or cost grounds. We test whether Step Law transfers to small language models. We consider three outcomes: H1, the original coefficients work directly; H2, the power-law form holds but

Read original source ↗ Open in workspace

recordType

paper

region

Global