SOURCE-LINKED INTELLIGENCE

MeRoTune: RoPE-Safe Merging with a Tunable Dial

arXiv · AI, language, vision and robotics · article · Sep 7, 2026 · UTC

When you merge two fine-tuned models from the same base checkpoint by simply averaging their weights, you implicitly assume their attention subspaces are still aligned. Recent work attempts to fix misalignments by learning an invertible correction matrix, $M$, for each model's query and key projections. This correction cancels out---using $M$ on the query side and $M^{-T}$ on the key side---right before the dot product. However, this cancellation is only exact if nothing sits between the projection and the dot product. In reality, almost all modern open-weight language models put a rotary posi

Read original source ↗ Open in workspace

recordType

paper

region

Global