SOURCE-LINKED INTELLIGENCE

Ranked by the Matcher: A Reproducibility Audit of Knowledge Graph Extraction from Threat Reports

arXiv · AI, language, vision and robotics · article · Sep 1, 2026 · UTC

Security teams and researchers choose knowledge-graph extraction tooling for threat reports on the strength of published triple-F1 scores, yet those scores depend on how predicted triples are matched to gold annotations. We could reimplement the stated matching rule for only five of twelve inspected systems. Re-scoring ten system outputs on shared documents under eight protocols reverses eleven of forty-five pairwise orderings; one fixed prediction set spans 0.16--0.70 F1. On an external, human-adjudicated set, no mechanical matcher---lexical, embedding, or entailment---agrees with the reviewe

Read original source ↗ Open in workspace

recordType

paper

region

Global