SOURCE-LINKED INTELLIGENCE

Judging a Review by its Cover: A Reliability Analysis of LLM-based Peer Review Evaluation Metrics

arXiv · AI, language, vision and robotics · article · Sep 20, 2026 · UTC

Peer-review evaluation is increasingly being automated with LLM-as-a-judge metrics, but this creates a measurement risk. A review may receive a high score because it is fluent, organized, and polished, rather than because it provides a strong evaluation of the paper. This risk is especially important in AI-assisted reviewing, where reviewers may use LLMs to improve clarity or presentation while preserving the underlying judgments. We propose a statistical framework for testing whether peer-review evaluation metrics capture substantive review quality beyond surface-level linguistic form. The fr

Read original source ↗ Open in workspace

recordType

paper

region

Global