SOURCE-LINKED INTELLIGENCE

MIGU: Multimodal Instruction Grounding under Uncertainty for Manipulation Planning

arXiv · AI, language, vision and robotics · article · Sep 21, 2026 · UTC

Understanding natural human instructions is crucial for deploying robots in human-centric environments. We study multimodal instruction grounding, where language and gesture provide complementary but uncertain cues. We present MIGU, a modular framework that combines semantic and geometric evidence into a unified grounding belief and connects it to manipulation planning. MIGU constructs a 3D geometric likelihood by propagating viewing-direction and depth uncertainty through eye-finger geometry while accounting for hand-direction estimation error. A vision-language model (VLM) provides semantic

Read original source ↗ Open in workspace

recordType

paper

region

Global