SOURCE-LINKED INTELLIGENCE

Measuring Optimal Transport in Transformer Depth

arXiv · AI, language, vision and robotics · article · Sep 1, 2026 · UTC

A transformer carries each token's state from layer to layer, and the whole vocabulary carried together forms a cloud that moves with depth. We ask whether a trained network moves this cloud the way optimal transport would: at the cheapest cost, and along the map that pairs each token with its optimal destination. We measure both on Pythia-160m and Pythia-410m, with an exact assignment between consecutive layer clouds, a measured sampling floor, calibration on couplings known to be optimal, and a split of the cost into the common shift of the cloud and the token-specific moves. At the last lay

Read original source ↗ Open in workspace

recordType

paper

region

Global