SOURCE-LINKED INTELLIGENCE

QUORUM: QUality-Optimized Routing Using Multiple annotators

arXiv · AI, language, vision and robotics · article · Aug 28, 2026 · UTC

Data annotation remains a central bottleneck in natural language processing, requiring human effort to obtain high-quality labels at scale. While Large Language Models (LLMs) offer a fast and cost-effective alternative, their reliability is highly instance-dependent: they perform well on simple inputs but often fail on examples requiring nuanced reasoning or contextual understanding. In this work, we address this challenge with QUORUM (QUality-Optimized Routing Using Multiple annotators), a budget-aware routing framework that dynamically assigns each instance to human or LLM annotators under a

Read original source ↗ Open in workspace

recordType

paper

region

Global