SOURCE-LINKED INTELLIGENCE

What Does an LLM-Agent Leaderboard Rank Actually Compare?

arXiv · AI, language, vision and robotics · article · Sep 7, 2026 · UTC

An LLM-agent leaderboard invites a familiar inference: an agent ranked above another is the better agent. Public evaluation logs may not support that conclusion when systems differ in task mixture, label source, release detail, or cost rule. We study what leaderboard scores estimate and when they justify pairwise superiority conclusions. Our estimand-aware pairwise procedure states the comparison target and measurement source, checks common support, and evaluates the supported difference using a stated uncertainty rule and practical margin. Controlled checks evaluate the decision labels under

Read original source ↗ Open in workspace

recordType

paper

region

Global