SOURCE-LINKED INTELLIGENCE

SPO++: Stream-Aligned Policy Optimization for Asynchronous Agentic RL

arXiv · AI, language, vision and robotics · article · Aug 25, 2026 · UTC

Group-relative reinforcement learning waits for sibling rollouts of the same prompt, which is costly for long and variable tool-use trajectories. Single-stream Policy Optimization (SPO) removes this dependency with a persistent prompt-level value estimate, but its recipe whitens one advantage per trajectory before optimizing a token-mean actor loss. We show that trajectory centering generally does not center the token-weighted quantity consumed by the actor, and fix the mismatch by standardizing terminal-outcome advantages under the action-token measure. We additionally organize prompt evidenc

Read original source ↗ Open in workspace

recordType

paper

region

Global