SOURCE-LINKED INTELLIGENCE

Do Dynamic Routers Need Memory? HeRo: History-Aware Routing for Efficient LLM Inference

arXiv · AI, language, vision and robotics · article · Sep 8, 2026 · UTC

Dynamic layer routing reduces the inference cost of Large Language Models (LLMs) by learning to skip layers for individual tokens. Existing methods, however, treat each routing decision as a local operation conditioned solely on the current hidden state which is a formulation that overlooks the sequential, path-dependent nature of routing across depth: earlier decisions shape the representations seen by downstream routers, and the layer-usage objective couples all decisions jointly. We propose History-Aware Routing (HeRo), a dynamic routing framework that resolves this mismatch by introducing

Read original source ↗ Open in workspace

recordType

paper

region

Global