SOURCE-LINKED INTELLIGENCE

A Hub of Short Rows Inflates Intrinsic Dimension Estimation of Token Embeddings

arXiv · AI, language, vision and robotics · article · Aug 30, 2026 · UTC

A token-embedding table holds a hub of short rows near its origin, and we show that this cluster biases what nearest-neighbor intrinsic-dimension (ID) estimators report. Because of the concentration of measure, a token is closer to the central cluster than to any other token, so its first two neighbors are both hub rows at nearly the same distance. As a result, the ID estimators such as TwoNN return a dimension far above the real ID. Measured one token at a time, dimension is a heavy-tailed distribution. Measured on the full vocabulary, it grows with the model's parameter count. However, when

Read original source ↗ Open in workspace

recordType

paper

region

Global