SOURCE-LINKED INTELLIGENCE

FlashBoB: I/O-Efficient Exact Backward-over-Backward for Softmax Attention

arXiv · AI, language, vision and robotics · article · Sep 21, 2026 · UTC

Transformer models built on the attention mechanism have become a central building block in modern deep learning, yet softmax attention remains a major bottleneck for long-context workloads. While FlashAttention makes the forward and first backward passes I/O-efficient, it does not support backward-over-backward (BoB), which enables exact differentiation through the backward pass for applications such as second-order optimization, test-time training, gradient-based memory, and meta-learning. Existing BoB implementations either materialize large intermediate tensors or exhaust GPU memory at lon

Read original source ↗ Open in workspace

recordType

paper

region

Global