SOURCE-LINKED INTELLIGENCE

Pinocchio: Fast Uncertainty Estimates for Black-Box Language Models

arXiv · AI, language, vision and robotics · article · Sep 21, 2026 · UTC

In high-stakes decision-making applications of large language models (LLMs), practitioners require not only accurate LLMs but also uncertainty estimates for their predictions. Existing approaches to uncertainty estimation for LLMs require access to log-probabilities output by the model or require fine-tuning access. However, many industrial LLM products use closed-source API models, and many such API models like GPT do not return log-probabilities and may not allow fine-tuning. We introduce Pinocchio, an external calibrator that estimates the correctness of responses from black-box API models.

Read original source ↗ Open in workspace

recordType

paper

region

Global