SOURCE-LINKED INTELLIGENCE

Beyond F1: Evaluating Coverage and Failure Recovery in AI Model Security Scanners

arXiv · AI, language, vision and robotics · article · Aug 27, 2026 · UTC

Static scanners are increasingly used to identify executable or otherwise unsafe content in machine- learning artifacts, yet conventional evaluation metrics characterize only cases where a scanner yields a usable security judgment. We evaluate ModelScan, ModelAudit, and Fickling using a controlled, artifact-backed benchmark on a synthetic corpus of 170 Pickle and PyTorch focused artifacts across 145 specimen families, 135 of which have binary security ground truth and 10 of which are intentionally malformed without labels. We explicitly distinguish non-N/A coverage, analysis completion, defini

Read original source ↗ Open in workspace

recordType

paper

region

Global