SOURCE-LINKED INTELLIGENCE

Multi-Head Self Attention is a Parameter Identification Mechanism

arXiv · AI, language, vision and robotics · article · Sep 1, 2026 · UTC

We prove that a multi-head scaled dot product attention can be viewed as a parameter identification strategy. The ratio of unidentified parameters to the total number of parameters scales like the reciprocal of the number of heads ($1/2 \to 1/(2H)$), meaning models with more heads are structurally more identified. A subtle side effect of the mathematics observation that attention can never be fully identified. Similarly we also show that some bias terms can have no effect on softmax-based attention layers in both the single- and multiple-head settings, though this is mostly a curiosity that sh

Read original source ↗ Open in workspace

recordType

paper

region

Global