SOURCE-LINKED INTELLIGENCE

Complex KDA: Understanding and Enhancing the Expressivity of Kimi Delta Attention

arXiv · AI, language, vision and robotics · article · Sep 21, 2026 · UTC

Linear RNNs based on the delta-rule enable efficient sequence modeling, but their linear updates with a low-rank correction constrain their expressivity. Prior work has shown that composing two delta-rule transitions in a single recurrent update can model a 2D rotation, but this increases the rank and the cost of the updates compared to a single transition. We show that Kimi Delta Attention (KDA) can realize 2D rotations by combining a single delta-rule transformation with a second reflection supplied by its channel-wise gate. This requires extending the parameter ranges of KDA by combining tw

Read original source ↗ Open in workspace

recordType

paper

region

Global