SOURCE-LINKED INTELLIGENCE

When Less is More: Understanding When Token Filtering Helps and Fails in AI-generated Text Detection

arXiv · AI, language, vision and robotics · article · Aug 30, 2026 · UTC

The rapid advancement of large language models (LLMs) has made AI-generated text detection increasingly critical. Existing zero-shot detectors assume that more token-level evidence leads to more reliable detection. However, our empirical study challenges this consensus: fewer tokens sometimes work better, retaining only 40% can yield optimal performance, yet this benefit is not universal. Using the Entropy Gap Score (EGS), we introduce top-$k$ cumulative probability filtering as a diagnostic probe. Across three representative settings, filtering exhibits strikingly different behaviors. We anal

Read original source ↗ Open in workspace

recordType

paper

region

Global