SOURCE-LINKED INTELLIGENCE

Optimizers for Diffusion Models: A Controlled Benchmark

arXiv · AI, language, vision and robotics · article · Sep 19, 2026 · UTC

Discrete diffusion models now match autoregressive language models on several benchmarks, while the question of how best to train them has received far less attention: the optimizer is inherited from one paper to the next and never compared. New optimizers, meanwhile, are validated almost exclusively on autoregressive pretraining, a different objective on a different loss surface. We present a controlled optimizer benchmark across four diffusion formulations, to our knowledge the first for discrete diffusion: seven optimizers (AdamW, Lion, Muon, SOAP, MARS, MARS-M, Schedule-Free) on masked dif

Read original source ↗ Open in workspace

recordType

paper

region

Global