SOURCE-LINKED INTELLIGENCE

Multi-Expert Conformal Risk Control for Pairwise LLM Judging in Open-Ended Dialogue

arXiv · AI, language, vision and robotics · article · Aug 27, 2026 · UTC

In this paper, we explore multi-expert Conformal Risk Control (CRC) algorithms for pairwise LLM-as-a-Judge evaluation in open-ended dialogue. Our core insight is that multi-expert aggregation offers a complementary remedy to CRC: whereas CRC controls risk at the decision threshold through abstention, aggregation sanitizes the scoring function at its source. Guided by this, we first design two multi-expert CRC methods: Score Averaging and Decision Voting, which aggregate at the score and decision levels, respectively. While both strategies outperform single-expert methods on homogeneous expert

Read original source ↗ Open in workspace

recordType

paper

region

Global