SOURCE-LINKED INTELLIGENCE

Compressing Long Context into Answer-Aligned Memory Embeddings for LLM Inference

arXiv · AI, language, vision and robotics · article · Sep 22, 2026 · UTC

Large language model (LLM) inference is constrained by the quadratic scaling of self-attention and the linear scaling of the KV cache, increasing latency, energy consumption, and GPU memory demand as context length scales. Existing soft-compression methods either lack query-guided memory selection at inference time, train without answer-targeted supervision, or couple compression tightly to a specific decoder architecture. We propose a Context-to-Answer-Aligned Memory Compression (CMC) framework, which compresses long input contexts into compact Context Memory Embeddings (CMEs) aligned to any

Read original source ↗ Open in workspace

recordType

paper

region

Global