SOURCE-LINKED INTELLIGENCE

Semantic Head Specialization Guides Hybrid ViT Attention for Multimodal LLMs

arXiv · AI, language, vision and robotics · article · Aug 28, 2026 · UTC

Hybrid attention dominates frontier LLMs, yet Vision Transformers (ViTs) in multimodal LLMs lack a satisfactory hybrid design, with no consensus on why certain attention patterns work better. To fill this gap, we study ViT attention heads and find they differentiate into object- and background-specialist roles, a pattern most pronounced under full attention; we call this Semantic Head Specialization (SHS). We propose SHS-Index to quantify this specialization, show that it distinguishes full-attention from chunk-window ViTs, and find that it strongly tracks downstream benchmark performance. We

Read original source ↗ Open in workspace

recordType

paper

region

Global