SOURCE-LINKED INTELLIGENCE

Efficient Benchmarking in Production: A Study of an Evolving LLM Agent

arXiv · AI, language, vision and robotics · article · Sep 18, 2026 · UTC

Production LLM agents are evaluated repeatedly as they evolve, but full agent benchmarks are costly to rerun. We study efficient recurring evaluation for a production analytics agent serving tens of thousands of monthly active users and report first-hand deployment experience. Using 574 historical runs of the production benchmark, split chronologically into calibration and held-out periods, we compare random sampling, historical caching, fixed representative subsets, and IRT-based adaptive testing. The results show that multidimensional 2PL adaptive testing achieves the best overall score fide

Read original source ↗ Open in workspace

recordType

paper

region

Global