SOURCE-LINKED INTELLIGENCE

Normalized Low-Rank Adaptation

arXiv · AI, language, vision and robotics · article · Aug 31, 2026 · UTC

While low-rank adaptation (LoRA) is widely used for parameter-efficient model adaptation, how to regularize its training dynamics for stable and effective optimization remains underexplored. Because LoRA initializes the up-projection to zero, its early optimization dynamics are largely governed by the down-projection. Building on this observation, we introduce Normalized Low-Rank Adaptation (NoRA), a simple yet effective method that normalizes the down-projection matrices during training. We further show that the same normalization can be applied only at initialization, improving standard LoRA

Read original source ↗ Open in workspace

recordType

paper

region

Global