SOURCE-LINKED INTELLIGENCE

Spectral Allocation: Why Muon Outperforms Adam, and How to Improve Muon

arXiv · AI, language, vision and robotics · article · Aug 26, 2026 · UTC

Orthogonal optimisers such as Muon can substantially accelerate large language model pretraining relative to Adam, yet the mechanism remains incompletely understood. We investigate this through an out-of-sample spectral probing analysis of Transformer loss landscapes. At checkpoints along real training trajectories, we decompose each momentum buffer into its singular directions and estimate the loss-optimal step size along each direction on held-out data. The resulting spectral profile is anisotropic yet stable across batches and training stages, and consistent across the optimisers and model

Read original source ↗ Open in workspace

recordType

paper

region

Global