SOURCE-LINKED INTELLIGENCE

GGUF-Metadata Prediction of Single-Sequence llama.cpp Throughput Across Three Systems

arXiv · AI, language, vision and robotics · article · Sep 14, 2026 · UTC

We predict single-sequence model throughput from GGUF metadata using roofline-shaped predictors with quantization-specific scale factors fitted on reference models. The scored cohort comprises 318 phase-depth measurements from 53 host-file configurations on two Apple M4 Max systems and an NVIDIA RTX 5080. On host-specific held-out sets of four, five, and two configurations, an active-parameter decode model obtains 13.1%, 14.4%, and 36.1% mean absolute percentage error (MAPE), versus 49.4%, 55.3%, and 51.9% when charging total parameters. Leave-one-host-out coefficients fitted on the other two

Read original source ↗ Open in workspace

recordType

paper

region

Global