SOURCE-LINKED INTELLIGENCE

HAP: Head-Adaptive Visual Token Pruning via Cross-Modal Alignment

arXiv · AI, language, vision and robotics · article · Aug 24, 2026 · UTC

Recent Vision-Language Models encode high-resolution images into long visual token sequences, incurring prohibitive prefill costs. To compress them, existing methods score each visual token by averaging text-to-visual attention uniformly across all heads, which assumes every head matches the query. However, our empirical analysis shows that misaligned heads dominate the average, amplifying background tokens and drowning out fine-grained cues. To address this, we propose PAQ (Prompt-Grounded Attention Quality), a metric quantifying how well each head aligns the prompt with image regions. Built

Read original source ↗ Open in workspace

recordType

paper

region

Global