SOURCE-LINKED INTELLIGENCE

Seeing Through Conflicts: Improving Instruction Hierarchy Alignment in Vision-Language Models

arXiv · AI, language, vision and robotics · article · Sep 3, 2026 · UTC

Instruction hierarchy (IH) alignment teaches language models to prioritize higher-level instructions when inputs conflict. While studied primarily in text-only settings, vision-language models (VLMs) introduce new challenges for IH: instructions may be embedded in images, split across modalities, visually transformed, or encountered during agentic tasks. Positing multimodal IH alignment as a reasoning problem, we train VLMs using reinforcement learning with rule-based rewards, comparing text-only, image-only, and mixed-modality supervision. We find that text-only IH training partially transfer

Read original source ↗ Open in workspace

recordType

paper

region

Global