SOURCE-LINKED INTELLIGENCE

On attention heads and bilinear forms

arXiv · AI, language, vision and robotics · article · Sep 19, 2026 · UTC

We study the symmetric and antisymmetric parts of bilinear forms in the attention heads of trained large language models. We introduce an orthogonally invariant profile map from real bilinear forms to a three-dimensional simplex and observe that profiles of trained bilinear forms accumulate near profiles of rank-one bilinear forms. We prove that the symmetric part of a bilinear form in an attention head is the sum of a hyperbolic form and a zero form for a Zariski-dense subset of query-key matrices.

Read original source ↗ Open in workspace

recordType

paper

region

Global