SOURCE-LINKED INTELLIGENCE

FlashLoop: Fast and Memory-Efficient Looped Transformers via Lazy Updates

arXiv · AI, language, vision and robotics · article · Sep 24, 2026 · UTC

Looped Transformers have attracted substantial attention as a parameter-efficient approach to increasing computational depth through repeated application of shared Transformer blocks. However, their practical advantages over conventional Transformers remain under debate: each additional loop incurs another Transformer pass and requires caching another set of KV states, causing inference FLOPs and KV-cache memory to grow continuously with loop depth. This overhead becomes particularly severe at large loop counts and long context, preventing the parameter efficiency of Looped Transformers from t

Read original source ↗ Open in workspace

recordType

paper

region

Global