SOURCE-LINKED INTELLIGENCE

The Approximation Rank of Softmax Attention: Sharp Geometric Laws and Robust Interaction Dimension

arXiv · AI, language, vision and robotics · article · Aug 28, 2026 · UTC

Which geometry controls the rank complexity of normalized softmax attention? We study maximum-row-$\ell_1$ approximation rank, exactly the least unrestricted rank preserving every bounded vector-valued output. Two sharp worst-case laws isolate support geometry: for fixed $d$ and error $\varepsilon$, spherical self-attention has rank $Θ_{d,\varepsilon}(\min\{n,(1+β)^{(d-1)/2}\})$, while full-ball geometry adds one radial degree and, for $β\geβ_0(d,\varepsilon)$ and $n\ge C_d e^{β/8}$, gives $Θ_{d,\varepsilon}(β^{d/2})$. For a fixed head, row-softmax quotients out row-scalar logit directions: th

Read original source ↗ Open in workspace

recordType

paper

region

Global