SOURCE-LINKED INTELLIGENCE

Decomposing LLM-Judge Uncertainty to Target Expert Labels

arXiv · AI, language, vision and robotics · article · Sep 6, 2026 · UTC

An LLM judge evaluates outputs at scale. Experts should label only where it is least sure. Its natural escalation signal conflates two uncertainties: aleatoric, real disagreement in the expert pool, which labels cannot reduce, and epistemic, the judge's ignorance, which labels do reduce. A small Bayesian model separates them: a regression on labels already collected learns how far to trust a black-box judge's prediction. Both components follow as simple formulas, with no sampling or further judge calls. The components isolate on a real LLM judge against exactly known truth, and stated confiden

Read original source ↗ Open in workspace

recordType

paper

region

Global