SOURCE-LINKED INTELLIGENCE

Context Staircase: Signature-Aligned Dynamics of Token Embeddings under Small Initialization

arXiv · AI, language, vision and robotics · article · Aug 31, 2026 · UTC

Token embeddings are the basic representational units that connect discrete tokens with continuous computation in language models. Although modern language models learn embeddings from random initialization through gradient-based training, the dynamical mechanism by which meaningful embedding structures emerge remains unclear. In this work, we identify that the evolving embedding structures are closely related to token-conditioned label and contextual distributions, which we formalize as probability signatures. We observe a progressive learning process, which we term Context Staircase: embeddi

Read original source ↗ Open in workspace

recordType

paper

region

Global