SOURCE-LINKED INTELLIGENCE

How Reproducible Are Evaluation Conclusions? A Self-Audit of LLM-Inferred Prompt Structure

arXiv · AI, language, vision and robotics · article · Sep 24, 2026 · UTC

Evaluations of LLM systems routinely average over small prompt sets and report models as a ranked table. We ask how much confidence such a table deserves, using LLM-based prompt-structure inference as the case study: eight open model variants across five families and 8B to 675B parameters, caching disabled, 293 raw intermediate representations persisted. The measured phenomenon is unstable to begin with. Identical calls do not reliably recover identical structure, with mean node-set Jaccard from 0.39 to 0.96 and 72% of prompt-model cells never node-set-perfect. Auditing the evaluation weakens

Read original source ↗ Open in workspace

recordType

paper

region

Global