SOURCE-LINKED INTELLIGENCE

Scaled Idempotence in Transformer Attention: Paired OV Geometry and Shared-Value Algebras

arXiv · AI, language, vision and robotics · article · Sep 1, 2026 · UTC

We identify a recurrent algebraic regularity in Transformer attention: a sparse subset of effective OV operators $T=OV^\top$ nearly closes under composition, $T^2\approxαT$. Across six pretrained endpoints spanning 2.8B--235B parameters, 3.98--8.00% of heads reach squared closure alignment $\mathcal{P}\geq0.9$, while no matched within-layer O/V mismatch does. An exact principal-coordinate factorization, $T=Q_OKQ_V^\top$ and $T^2=Q_O(KDK)Q_V^\top$, separates within-support transport from read--write return geometry. Across all 7,304 heads in nine MHA/GQA models, scrambling only the orientation

Read original source ↗ Open in workspace

recordType

paper

region

Global