SOURCE-LINKED INTELLIGENCE

Comparing Latent Concept Formation in State Space Models and Transformers via Sparse Autoencoders

arXiv · AI, language, vision and robotics · article · Sep 21, 2026 · UTC

The quadratic scaling of Transformer self-attention has driven the adoption of sub-quadratic Selective State Space Models (SSMs) like Mamba, which compress past context into a fixed-size recurrent hidden state. This strict informational bottleneck raises a foundational question for mechanistic interpretability: do SSMs and Transformers learn fundamentally distinct latent representations? In this work, we employ Sparse Autoencoders (SAEs) to conduct a large-scale, feature-level correspondence analysis between Mamba-130m and Pythia-70m over a 10-million token corpus. Contrary to hypotheses predi

Read original source ↗ Open in workspace

recordType

paper

region

Global