SOURCE-LINKED INTELLIGENCE

Optimizer Memory Schedules for Outscaling the Overtraining Axis

arXiv · AI, language, vision and robotics · article · Sep 4, 2026 · UTC

We investigate how optimizers scale across the overtraining axis and show that relative optimizer performance and optimal hyperparameters change substantially with training horizon. In particular, we study how matrix-preconditioned methods (Muon and SOAP) and a momentum-scheduled method (ADANA) scale relative to AdamW. We compare these four optimizers across models from 51M to 253M parameters and overtraining (OT) factors from 1x to 256x, sweeping the base learning rate at every setting. The preferred learning rate schedule can reverse across the overtraining axis, the best weight decay coeffi

Read original source ↗ Open in workspace

recordType

paper

region

Global