SOURCE-LINKED INTELLIGENCE

Virtual Encoders in Multimodal Transformers

arXiv · AI, language, vision and robotics · article · Sep 22, 2026 · UTC

Multimodal language models traditionally rely on dedicated perceptual encoders to construct task-usable representations. More integrated architectures have recently emerged, which instead expose the shared transformer to lightly projected patches, audio frames, or discrete visual tokens. Where does this encoding happen when such representations are not provided? We find that the transformer can internalize this missing computation, constructing task-usable perceptual representations within its own early-to-middle layers before the downstream language model. We call this computational structure

Read original source ↗ Open in workspace

recordType

paper

region

Global