SOURCE-LINKED INTELLIGENCE

Abstention and Noise Filtering: Two Missing Primitives of Softmax Attention

arXiv · AI, language, vision and robotics · article · Sep 18, 2026 · UTC

Gating the value pathway of attention reportedly improves language model pretraining, and prior studies disagree on why. We argue and provide experimental evidence that such gates supply two different things that softmax attention lacks: abstention and noise filtering. The first is abstention, which allows an attention head to output nothing, bypassing the requirement that attention weights must sum to one. The second is noise filtering, which allows the value pathway of an attention head to suppress interference from superposed features in the residual stream. In our experiments in matched mo

Read original source ↗ Open in workspace

recordType

paper

region

Global