SOURCE-LINKED INTELLIGENCE

Tensor Decomposition of Transformer Key-Value Caches: Spectral Structure and Format Comparison

arXiv · AI, language, vision and robotics · article · Sep 23, 2026 · UTC

The key-value (KV) cache of autoregressive transformers can be viewed as a fourth-order tensor spanning attention heads, tokens, features, and grouped layers. We measure the singular-value spectra of all four mode unfoldings on Mistral-7B-v0.3 and LLaMA-2-13B and compare four standard tensor decompositions: Tucker, CP, tensor train, and t-SVD, at matched storage. The spectra partition the four axes into two classes. The token and feature modes carry low-rank structure, particularly for keys. The head and layer modes are nearly full-rank and resist compression at any practical error level. Amon

Read original source ↗ Open in workspace

recordType

paper

region

Global