SOURCE-LINKED INTELLIGENCE

SpeechGym: An Audio-Native Gym for Training Voice Agents via Reinforcement Learning

arXiv · AI, language, vision and robotics · article · Aug 26, 2026 · UTC

Voice agents must call tools and hold multi-turn dialogue entirely through speech, yet the dominant paradigm trains them in text. Existing frameworks either cascade TTS and ASR around a proprietary voice API, where gradients cannot flow and per-call cost makes on-policy reinforcement learning prohibitive, or stay in text: they measure voice agents but cannot improve them. We present SpeechGym, an audio-native agentic environment in which two omni-modal models converse in native audio, with no external ASR or TTS and no API boundary, over the unmodified tasks, tools and success check of an esta

Read original source ↗ Open in workspace

recordType

paper

region

Global