SOURCE-LINKED INTELLIGENCE

When Tokenization is Secretly Output Supervision

arXiv · AI, language, vision and robotics · article · Sep 1, 2026 · UTC

Tokenization in language models is treated by default as an input preprocessing decision. We argue that this framing is incomplete: in autoregressive models, tokenizer granularity determines what the model must resolve in a single forward pass, and therefore the supervision signal it receives. This affects both the difficulty of the learning problem and the representations that emerge inside the model. We test this in a controlled experiment on numeric reasoning with a novel decoupling of input and output tokenization. As the output supervision view predicts, differences in task performance, t

Read original source ↗ Open in workspace

recordType

paper

region

Global