SOURCE-LINKED INTELLIGENCE

Delayed Optimizer-State Transport Shapes Short-Horizon Training Decisions

arXiv · AI, language, vision and robotics · article · Aug 25, 2026 · UTC

Adaptive optimizers retain gradient history in moment variables, allowing a local change in loss weighting to alter later updates. We examine whether this delayed transport is large enough to change prospective short-horizon decisions. On committed future-minibatch sequences, we differentiate eight-step AdamW trajectories through the complete model--optimizer state and select exposure-matched Math--Code loss schedules before independent evaluation. Across 12 unused 0.3M Transformer histories, full transport lowers token-disjoint loss relative to an optimizer-aware immediate derivative in 10/12

Read original source ↗ Open in workspace

recordType

paper

region

Global