SOURCE-LINKED INTELLIGENCE

trajectory-judge: What Outcome-Only LLM Judges Miss on Agent Trajectories

arXiv · AI, language, vision and robotics · article · Aug 29, 2026 · UTC

Outcome-only evaluation is the production default for LLM agents: show a judge the request and the final reply and ask whether it was handled well. The metric is structurally blind to an agent that reaches the right answer the wrong way. We measure that blind spot where ground truth is known by construction: a deterministic tool-using support-desk environment, a scripted oracle policy that always solves it, and a fault injector that breaks exactly one thing at a known step, stratifying faults by whether the customer-visible outcome survived (silent) or not (loud). Five judges (programmatic rul

Read original source ↗ Open in workspace

recordType

paper

region

Global