SOURCE-LINKED INTELLIGENCE

The Hidden Ratio in Adam: Stable Structure, Compression, and Sign Dynamics

arXiv · AI, language, vision and robotics · article · Sep 28, 2026 · UTC

Adam is the default optimizer for training modern deep neural networks, yet its adaptive behavior remains poorly understood due to the complex interaction between its first- and second-moment exponential moving averages (EMAs). We study Adam in the tied-$β$ regime, where the two EMA decay rates are equal, and show that its adaptive dynamics can be expressed through a transformed ratio with approximately scale-stable behavior. Empirically, this transformed ratio exhibits a stable, heavy-tailed distribution across tasks, model scales, and training stages, in contrast to the variability of raw mo

Read original source ↗ Open in workspace

recordType

paper

region

Global