SOURCE-LINKED INTELLIGENCE

Localize-Then-Decide Guarantees for LLM Judgments

arXiv · AI, language, vision and robotics · article · Aug 26, 2026 · UTC

Large language models (LLMs) are increasingly used as evaluators to assess output quality and preference alignment, yet providing reliable guarantees of agreement with human judgments remains challenging. Recent work introduces confidence-thresholding methods that provide such guarantees for pairwise comparisons, relying on the assumption that higher estimated confidence implies lower disagreement risk with humans. However, this assumption can break down when the number of candidate responses increases, since distributing probability mass across many alternatives can distort confidence estimat

Read original source ↗ Open in workspace

recordType

paper

region

Global