SOURCE-LINKED INTELLIGENCE

Selective Agent Guidance via Entropy: Learning Autonomous Policies from Imperfect VLM Teachers

arXiv · AI, language, vision and robotics · article · Sep 1, 2026 · UTC

Vision-Language Models (VLMs) provide useful priors for interactive decision-making, but using them directly as policies is expensive and brittle: they must be queried at every step, do not improve from environment interaction, and can repeat systematic errors. We study how to learn a cheap autonomous policy from an online, expensive, and imperfect but informative VLM teacher. We propose SAGE (Selective Agent Guidance via Entropy), a framework that queries a VLM only when the learner is uncertain, executes the suggested action during training, and distills guidance into a lightweight Reinforce

Read original source ↗ Open in workspace

recordType

paper

region

Global