SOURCE-LINKED INTELLIGENCE

Every Layer Counts: An Exponential $L_2$ Depth Hierarchy for ReLU Networks

arXiv · AI, language, vision and robotics · article · Aug 24, 2026 · UTC

We prove a depth hierarchy for ReLU neural networks in which every additional ReLU layer can save exponentially many neurons. For all $k\geq2$, we construct a globally $[0,1]$-valued, $1$-Lipschitz function realized by a depth-$(k+1)$ network of width $\mathcal{O}(d^4)$, whereas any depth-$k$ network with unrestricted weights and width at most $\frac{2^d}{2d(k-1)}$ has squared $L_2$ error at least $1/24$ under an absolutely continuous distribution supported at exponential distance from the origin. To the best of our knowledge, this is the first exponential hierarchy across all adjacent fixed d

Read original source ↗ Open in workspace

recordType

paper

region

Global