SOURCE-LINKED INTELLIGENCE

One Rate Is Not Enough: Adaptive Anisotropic Learning Rates for LoRA Fine-Tuning

arXiv · AI, language, vision and robotics · article · Sep 5, 2026 · UTC

Low-rank adaptation (LoRA) has become the standard for parameter-efficient fine-tuning of large language models. Most LoRA variants follow a uniform-LR convention, applying a single global learning rate across every rank-one component of every adapter. We show that this convention overlooks substantial within-module heterogeneity, where the rank-one components of a LoRA adapter update at highly uneven rates and low-velocity modules converge to concentrated singular spectra that underutilize the nominal rank budget. To address this, we propose an adaptive anisotropic learning-rate model that as

Read original source ↗ Open in workspace

recordType

paper

region

Global