SOURCE-LINKED INTELLIGENCE

SA-Bench: Evaluating Semantic Alignment in LLM-Based Paper Reproduction

arXiv · AI, language, vision and robotics · article · Aug 25, 2026 · UTC

LLM agents can generate paper reproduction code, yet often produce scientifically unfaithful implementations. We define this failure mode as semantic drift, where generated code silently diverges from the paper's specifications. We introduce SemanticAlign-Bench(SA-Bench), a diagnostic benchmark covering 30 papers from ICLR, ICML and NeurIPS 2025. For each paper, we decompose its specifications into atomic and verifiable implementation claims, which we call Semantic Alignment Units (SAUs) and evaluate repositories along four diagnostic dimensions spanning numerical, methodological, protocol and

Read original source ↗ Open in workspace

recordType

paper

region

Global