SOURCE-LINKED INTELLIGENCE

Almost Free State Prediction Separation

arXiv · AI, language, vision and robotics · article · Sep 3, 2026 · UTC

State--prediction separation (SPS) relieves a language model's hidden state of two competing burdens---summarizing the context and predicting the next token---by splitting the forward pass into a state stream and a prediction stream. The separation works, but it is expensive: the prediction stream is a second pass over the whole backbone, costing $\sim$1.9$\times$ the pretraining FLOPs, and even more in terms of wall-clock time when using a flexible attention mask. This paper makes state--prediction separation almost free. We take the separation to its limit with a free pause token: a predicti

Read original source ↗ Open in workspace

recordType

paper

region

Global