SOURCE-LINKED INTELLIGENCE

Rethinking Human-Aligned Evaluation: An Analysis of Semantic Metrics Beyond WER

arXiv · AI, language, vision and robotics · article · Sep 18, 2026 · UTC

Word Error Rate (WER), the most commonly used metric for Automatic Speech Recognition (ASR), treats every lexical deviation from the reference as equally costly, regardless of whether it changes meaning. This raises the question: does WER actually track how humans judge ASR transcript quality? We introduce HATS-en, an English dataset for human-centered ASR evaluation. Using this dataset, we benchmark lexical metrics against several configurations of BERTScore and SemDist, varying the language model, layer, and pooling strategy. We find that WER agrees least with human judgment among all metric

Read original source ↗ Open in workspace

recordType

paper

region

Global