SOURCE-LINKED INTELLIGENCE

Visual Attention Faithfulness in Vision-Language Models is Heterogeneous

arXiv · AI, language, vision and robotics · article · Sep 1, 2026 · UTC

Whether attention weights faithfully reflect model reasoning has been actively debated in NLP, yet this question remains largely unexplored for the visual modality in Vision-Language Models (VLMs). We address this gap through causal perturbation analysis on current VLMs, evaluating both the comprehensiveness and sufficiency gap of attention-ranked visual tokens. Our analysis reveals that visual attention faithfulness is heterogeneous, manifesting in three distinct processing modes: Faithful-Sufficient, where top-$k$ attention tokens are both necessary and sufficient for prediction; Faithful-Di

Read original source ↗ Open in workspace

recordType

paper

region

Global