SOURCE-LINKED INTELLIGENCE

Human-Anchored Factuality Evaluation with Strategic Annotation

arXiv · AI, language, vision and robotics · article · Aug 31, 2026 · UTC

LLM-based factuality judges provide scalable evaluation signals, but their metrics are often systematically biased relative to human judgments. We study human-anchored factuality evaluation under limited annotation budgets, where judge predictions on the full dataset are combined with human labels on a small selectively sampled subset to obtain statistically valid estimates. The efficiency of this approach depends critically on which examples receive human annotation: in factuality evaluation, judge-human misalignment is not driven solely by low confidence, but also by structured failure modes

Read original source ↗ Open in workspace

recordType

paper

region

Global