SOURCE-LINKED INTELLIGENCE

Think-Probe-Respond: Improving Large Language Models as Judges of Research Idea Novelty

arXiv · AI, language, vision and robotics · article · Aug 26, 2026 · UTC

Automated novelty judgment can accelerate scientific discovery by enabling efficient evaluation, refinement, and comparison of research ideas. While large language models are increasingly adopted for this task, we investigate a previously overlooked limitation in their judgment capabilities: despite generating reasoning rationales that closely mirror those of human experts, their final novelty judgments often diverge substantially. We demonstrate that this miscalibration stems from a systematic bias towards judging ideas as "medium novel". To mitigate this, we propose Think-Probe-Respond (TPR)

Read original source ↗ Open in workspace

recordType

paper

region

Global