SOURCE-LINKED INTELLIGENCE

Stiefel-AdamW: Geometry-Aware AdamW for Linear Factorization Blocks

arXiv · AI, language, vision and robotics · article · Sep 17, 2026 · UTC

A pervasive structural pattern in modern deep learning is the linear factorization block: a submodule of the form $W = BA$ in which two parameter matrices are multiplied directly, with no intervening nonlinearity. Such blocks appear in LoRA adapters, low-rank compressed layers, query-key products of self-attention, and share a common pathology: the factorization is non-unique, which can destabilize training and limit usable learning rates. Despite this, factorization blocks are typically optimized with standard Euclidean methods that ignore the underlying geometry. We introduce Stiefel-AdamW,

Read original source ↗ Open in workspace

recordType

paper

region

Global