SOURCE-LINKED INTELLIGENCE

XYBench: Can LLMs Respond Pragmatically to Queries with Misconceptions?

arXiv · AI, language, vision and robotics · article · Sep 6, 2026 · UTC

When non-expert users ask LLMs for assistance, their queries can often have misconceptions (e.g., "How do I parse XML with regex?"). In such cases, often referred to as the XY-problem, LLMs must identify the misconception ("regex are fragile") and meaningfully direct the user toward a pragmatic solution that will address the root problem implicit in the request ("use an XML parser"). We introduce XYBench, a benchmark of 8,115 such queries, drawn from technical (StackOverflow/StackExchange) and everyday (WikiHow and a manually-curated subset) domains. We design an evaluation paradigm that asses

Read original source ↗ Open in workspace

recordType

paper

region

Global