SOURCE-LINKED INTELLIGENCE

Calibration as a First-Class Criterion in LLM Evaluation

arXiv · AI, language, vision and robotics · article · Sep 22, 2026 · UTC

Calibration of language models -- the alignment between expressed or implicit confidence and empirical correctness -- is a well-studied subfield within NLP. Methods to measure it already exist. The problem is adoption: outside this subfield, NLP research regularly introduces new models, datasets, and benchmarks without checking whether the model's confidence scores are meaningful. We argue that this adoption gap is a major obstacle to trustworthy LLM evaluation. Miscalibration causes problems in two distinct areas: at deployment, where overconfident mistakes cause real harm, and inside the res

Read original source ↗ Open in workspace

recordType

paper

region

Global