SOURCE-LINKED INTELLIGENCE

The Geometry of Ignorance: LLMs Know When to Temper Bayesian Priors

arXiv · AI, language, vision and robotics · article · Sep 2, 2026 · UTC

What does a language model predict when it has few clues? The answer lurks in its unembedding geometry: a single direction of the unembedding matrix encodes the unigram distribution of the training corpus, which serves as the Bayesian prior the model falls back on when uncertain. This structure --- which we term the \emph{direction of ignorance} --- appears in all four model families examined (\texttt{Llama}, \texttt{Qwen}, \texttt{Gemma}, and \texttt{Pythia}), ranging from 0.4B to 405B parameters. Projecting the final prediction state onto this direction yields a per-token \emph{prior loading

Read original source ↗ Open in workspace

recordType

paper

region

Global