SOURCE-LINKED INTELLIGENCE

What the Window Does Not Contain: Auditing Provenance in a Document-Grounded Instability Benchmark

arXiv · AI, language, vision and robotics · article · Sep 5, 2026 · UTC

Ask a language model the same question about the same document twenty times, and it sometimes returns two different answers. We built Probity, a benchmark of 60 tasks and 470 items from real venture-financing filings, to measure how often this happens. Then we audited our own corpus and found a defect any excerpt-built benchmark can carry: items whose evidence is missing from the window of text the model is shown. The audit flags 36 items and separates two failures a single flag would conflate: evidence genuinely absent from the window and answers that must be computed from numbers the window

Read original source ↗ Open in workspace

recordType

paper

region

Global