SOURCE-LINKED INTELLIGENCE

Towards a mathematical theory of superposition

arXiv · AI, language, vision and robotics · article · Aug 27, 2026 · UTC

We develop a mathematical theory of superposition in neural networks using tools from frame theory and compressed sensing. In our model, a sparse binary vector \(x\) of active features is encoded through an overcomplete dictionary \(W\), and feature recovery is performed by applying \(\operatorname{ReLU}(W^\top W x+b)\) with an appropriate bias vector \(b\). We prove several recovery theorems for this model. In the random-support setting, we establish high-probability support recovery for nearly tight, low-coherence dictionaries, with guarantees when the expected sparsity is up to order \(d/\l

Read original source ↗ Open in workspace

recordType

paper

region

Global