What Does Multi-Agent LLM Debate Actually Change? A Layered Analysis of Disagreement and Answer Quality

2026-09-25T10:42:36.266Z

• title: A Layered Analysis of Disagreement And Answer Quality in Multi-Agent LLM Debate → What Does Multi-Agent LLM Debate Actually Change? A Layered Analysis of Disagreement and Answer Quality
• summary: Multi-agent debate, in which several LLMs exchange arguments before answering, is widely assumed to improve answer quality by surfacing genuine disagreement. That mechanism is rarely checked. We introduce four measurements: (A) the agreement a debater reports; (B) whether its reply text actually pushes back; (C) whether the position persists once the eliciting instruction is removed; and (D) for open-weight models, the stance response in the debater's own token log-probabilities. We evaluate three-model committees debating open-ended GlobalOpinionQA across 750 debates under three tones: friend → Multi-agent debate, in which several LLMs exchange arguments before producing an answer, is widely assumed to improve answer quality by surfacing genuine disagreement. That disagreement is hard to verify, and no single signal can settle it, so we organize the analysis around four questions: (A) does the debater say it disagrees; (B) does its reply text actually argue; (C) does the dissent survive once the tone instruction that produced it is removed; and (D) do the probabilities assigned to stance options change? We evaluate three-model committees on 50 open-ended GlobalOpinionQA questions und