SOURCE-LINKED INTELLIGENCE

Preserving What Matters: Semantic Scaffolds Beyond Saturation in Summarization Evaluation

arXiv · AI, language, vision and robotics · article · Sep 18, 2026 · UTC

Summarization ships in countless production systems, making model selection a routine decision that depends on measuring summary quality. Existing metrics struggle to support this: ROUGE captures only surface overlap, while LLM-as-judge scores saturate to near-identical values that fail to rank models effectively. We observe this saturation across three public datasets, two proprietary datasets, and multilingual settings. Motivated by this, we introduce Semantic Scaffold, an evaluation framework that extracts a hierarchical representation of facts, questions, and entity attributes from a sourc

Read original source ↗ Open in workspace

recordType

paper

region

Global