SOURCE-LINKED INTELLIGENCE

ClusterAttention: A training-free speedup of bidirectional attention

arXiv · AI, language, vision and robotics · article · Aug 27, 2026 · UTC

This paper introduces ClusterAttention, a general training-free speedup of bidirectional attention layers. Existing sparse attention methods either rely on structure in the input, such as order in language or spatial proximity in images, or use slow clustering processes amortized over several forward passes. ClusterAttention instead uses a fast recursive clustering method that adapts to the geometry of the keys and queries in each attention head to produce useful clusters. This method allows setting the size of the clusters arbitrarily. We utilize this by setting all clusters to be a fixed siz

Read original source ↗ Open in workspace

recordType

paper

region

Global