SOURCE-LINKED INTELLIGENCE

H-Spec: Parallel Speculative Decoding Without a Drafter-Side KV Cache

arXiv · AI, language, vision and robotics · article · Sep 21, 2026 · UTC

Speculative decoding losslessly accelerates large language model inference by having a lightweight draft model predict future tokens for verification by the target model. Recent block diffusion drafters further reduce drafting latency by predicting multiple tokens in parallel. However, existing block drafters project target hidden states at every input position into a separate drafter-side KV cache, incurring per-request memory and KV-write overhead that grow with concurrency; directly reusing target KVs in place removes this cache but fails to sustain draft quality throughout the block. We pr

Read original source ↗ Open in workspace

recordType

paper

region

Global