SOURCE-LINKED INTELLIGENCE

A Layer Importance Metric for Quantization Accounting for the Speed-Quality Trade-off in Autoregressive Models

arXiv · AI, language, vision and robotics · article · Aug 27, 2026 · UTC

Small language models (sLLMs) are nowadays hosted on devices with limited memory and computational budget. In an autoregressive setup, inference is memory-bandwidth bound: uniform quantization is often detrimental to such models, since their architecture has limited redundancies and only a few layers are not very sensitive to lower precision. We propose a composite metric that combines two orthogonal criteria: information retention (measured in terms of a normalized SQNR-based coefficient) and throughput gains (modeled using a roofline-based latency analysis). By profiling Gemma 3 1B, we find

Read original source ↗ Open in workspace

recordType

paper

region

Global