SOURCE-LINKED INTELLIGENCE

RoPE attention is an exact forward-pass gradient step with softmax intact

arXiv · AI, language, vision and robotics · article · Sep 6, 2026 · UTC

We derive an exact gradient-step representation of the RoPE-softmax forward pass. For every deterministic RoPE-softmax attention head with arbitrary affine projection weights, we construct a query-dependent effective matrix $ΔM_i$ satisfying $y_i = μ_i + u_i^\top ΔM_i$, where $μ_i$ is the uniform mean of the attended values and $u_i$ is the augmented query input. The construction applies the classical exponential divided difference $ρ= φ_1$ to retain the softmax exactly. Its positive coefficients give a unit gradient-step representation on a query-conditioned quadratic objective. The same func

Read original source ↗ Open in workspace

recordType

paper

region

Global