SOURCE-LINKED INTELLIGENCE

Muon Can Outperform Dedicated Continual Learning Methods

arXiv · AI, language, vision and robotics · article · Sep 21, 2026 · UTC

Continual learning with Low-Rank Adapters (LoRA) typically mitigates forgetting by penalizing the overlap between a new update and the accumulated past weights, which discourages certain update directions without controlling how an update distributes its energy over the ones that remain. We ask whether that restriction has to be task-aware, or whether a generic one supplied by the optimizer is enough. We train a plain incremental LoRA (IncLoRA) with Muon, which orthogonalizes each update, and compare it against O-LoRA and ELLA over five seeds and three task orders on the Standard CL Benchmark

Read original source ↗ Open in workspace

recordType

paper

region

Global