SOURCE-LINKED INTELLIGENCE

Distilling Sequential Computation in Transformer Language Models

arXiv · AI, language, vision and robotics · article · Sep 23, 2026 · UTC

Transformer language models process sequences token by token in an autoregressive manner, making growing contexts increasingly expensive. Yet many adjacent token spans are highly predictable or frequently occur as stable units, suggesting that their representations may be compressible. We introduce a method for distilling sequential computation by replacing spans of input tokens with collapsed representations, computed on the fly by a lightweight merge module. This module generates a single surrogate embedding from a sequence of static token embeddings that captures the functional role of the

Read original source ↗ Open in workspace

recordType

paper

region

Global