SOURCE-LINKED INTELLIGENCE

Can LLMs Reason About Runtime Behavior? A Repository-Level Dynamic Benchmark

arXiv · AI, language, vision and robotics · article · Sep 23, 2026 · UTC

Large language models (LLMs) are increasingly used in coding tasks, but their ability to reason about code execution remains unclear. Existing repository-level QA benchmarks mainly evaluate static code understanding and often rely on LLM-based evaluation, while execution-reasoning benchmarks are mostly limited to snippets or functions. We introduce SWE-Flux, a repository-level benchmark for dynamic execution reasoning containing 480 execution-grounded instances across 12 real Python repositories, with gold answers automatically harvested from instrumented test executions rather than written ma

Read original source ↗ Open in workspace

recordType

paper

region

Global