SOURCE-LINKED INTELLIGENCE
ENGINE: Endogenous Variational MultiScale Optimization for Zeroth-Order LLM Fine-Tuning
Zeroth-order (ZO) optimization enables memory-efficient large language model (LLM) fine-tuning, and recent methods incorporate curvature-aware strategies. However, in the high-dimensional parameter space of LLMs, these methods inherently suffer from exploding gradient variance and prohibitive costs. Conversely, confining updates to a low-dimensional subspace reduces variance but discards critical high-dimensional information, degrading performance. This creates a fundamental dilemma: full-space optimization is informative but has high variance, while subspace optimization has low variance but
Read original source ↗ Open in workspace
- recordType
- paper
- region
- Global
Evidence & attribution
- arXiv · AI, language, vision and robotics · 2026-11-02T15:31:38.000Z
First collected: 2026-11-03T08:23:14.856Z. This is not the publication date.