SOURCE-LINKED INTELLIGENCE

The Depth Flow of Token Representations Is Nonlinear and Does Not Descend Its Own Density

arXiv · AI, language, vision and robotics · article · Aug 30, 2026 · UTC

A token's representation is carried through the network layer by layer. The whole vocabulary carried together forms a flow. We fit this flow's equation of motion as a discrete Langevin model over corpus-mean trajectories of Pythia-160M and Pythia-410M, and score the predicted steps on held-out tokens. Linear maps are often used as cheap surrogates for a layer. The flow they summarize is not linear: a quadratic drift beats the linear linear map at every transition of both models, and the Kramers--Moyal estimator agrees wherever its neighborhoods stay local. We then characterize the flow further

Read original source ↗ Open in workspace

recordType

paper

region

Global