SOURCE-LINKED INTELLIGENCE

Spurious Advantage Hidden in GRPO

arXiv · AI, language, vision and robotics · article · Sep 3, 2026 · UTC

Group Relative Policy Optimization (GRPO) is widely studied for reinforcement learning with verifiable rewards, where its advantage estimator assigns each rollout a magnitude from within-group reward statistics. In the common case, this magnitude rewards rollouts that reach the correct answer through reasoning. Yet, an overlooked case shares the same surface: a rollout may land on it by guessing, and the formula still assigns a high magnitude, which we identify as the spurious advantage. This arises in three cases: bounded-answer tasks with a small candidate set; open-answer sets hosting bound

Read original source ↗ Open in workspace

recordType

paper

region

Global