SOURCE-LINKED INTELLIGENCE

oHC: Orthogonal Hyper-Connections on SO(4) via Quaternions

arXiv · AI, language, vision and robotics · article · Sep 2, 2026 · UTC

Hyper-Connections (HC) replace the single residual stream of a Transformer with $n$ parallel ones, mixing them at every layer with a learned $n \times n$ residual matrix. Leaving that matrix unconstrained places no limit on the factor by which the mixing step rescales the residual streams, and that factor compounds across layers, which destabilizes training. Manifold-constrained Hyper-Connections (mHC) address this by restricting the matrix to the doubly stochastic matrices. That caps the factor at one, so the mixing can no longer amplify any direction, but nothing bounds it from below. We pro

Read original source ↗ Open in workspace

recordType

paper

region

Global