SOURCE-LINKED INTELLIGENCE

Memory Attention

arXiv · AI, language, vision and robotics · article · Sep 23, 2026 · UTC

Language models typically construct attention values from contextual hidden states, even when some of their content may be reusable across contexts. We investigate whether token-indexed memory can replace the dedicated value projection when complemented by contextual information. We propose Memory Attention (MA), which forms values by combining layer-specific token memory with contextual keys. The memory supplies token-specific representations, while the keys preserve context dependence. At inference, normalization can be folded into the memory tables, reducing value construction to lookup and

Read original source ↗ Open in workspace

recordType

paper

region

Global