SOURCE-LINKED INTELLIGENCE

Transcribe, Translate, and Optimize: Joint Reward Learning for Speech Translation

arXiv · AI, language, vision and robotics · article · Sep 22, 2026 · UTC

In LLM-based speech translation, transcription-based chain-of-thought (CoT) suffers from a mismatch between reference transcripts used in supervised fine-tuning (SFT) and model-generated transcripts at inference. To address this, we propose joint recognition and translation fine-tuning via group relative policy optimization (GRPO). We score both transcripts and translations, with translation conditioned on model-generated transcripts, and compare three token advantage strategies. Using Qwen2.5-Omni-3B across four languages, we evaluate CoT against direct speech translation (Direct ST) under SF

Read original source ↗ Open in workspace

recordType

paper

region

Global