SOURCE-LINKED INTELLIGENCE

Language Models Can Control Their Own Attention

arXiv · AI, language, vision and robotics · article · Sep 2, 2026 · UTC

Language models spend most of their attention on a small fraction of context, yet they read the entire KV cache to find the few tokens that matter. If the user asks about a previous detail in a 1M-token conversation, global attention layers must scan the full context to generate each token of the reply. A prominent approach mitigates this cost by pre-selecting relevant tokens via lightweight proxy scores, but this extrinsic scoring still incurs O(N) per step. We take an intrinsic approach motivated by the simple question: wouldn't the model already know which parts of the context are relevant?

Read original source ↗ Open in workspace

recordType

paper

region

Global