SOURCE-LINKED INTELLIGENCE

Fine-Tuning of Transformer models with Frames

arXiv · AI, language, vision and robotics · article · Aug 26, 2026 · UTC

Parameter-Efficient Fine-Tuning (PEFT) strategies such as Low-Rank Adaptation (LoRA) are effective solutions for fine-tuning large-scale pre-trained models; however, their memory requirements scale with the size of the model, $\mathcal{O}(dr)$, where $d$ is the model's hidden dimension and $r$ is the rank. Our proposal, FrameFT, models the parameter update $ΔW$ with a sparse coefficient matrix in a Fusion Frame basis. Fusion Frames can be generated algorithmically and shared across model layers, enabling very efficient updates. Only the sparse coefficients of the basis expansion are stored/opt

Read original source ↗ Open in workspace

recordType

paper

region

Global