SOURCE-LINKED INTELLIGENCE

The First Token Is a Clue: Verbalizing Multi-Token Concepts from the J-lens

arXiv · AI, language, vision and robotics · article · Aug 31, 2026 · UTC

The Jacobian Lens (J-lens) is a recent tool for interpreting LLMs. It reads a hidden state as a ranked list of vocabulary tokens, leaving multi-token concepts without a representation of their own. The original J-lens work addresses this limitation with Template Lens, which precomputes vectors for a fixed phrase vocabulary, and Oracle Lens, which fine-tunes components to propose phrases and reconstruct phrase vectors. We ask whether multi-token concepts and their vectors can instead be recovered directly from J-lens and the frozen model. We find that the first token of a multi-token concept is

Read original source ↗ Open in workspace

recordType

paper

region

Global