SOURCE-LINKED INTELLIGENCE

SHIFT-LLM: Distribution Shift Correction in Depth-Pruned LLMs

arXiv · AI, language, vision and robotics · article · Aug 25, 2026 · UTC

Depth pruning removes entire Transformer blocks to reduce the inference cost of large language models, but disrupts the hidden-state distributions expected by downstream layers, leading to significant accuracy loss. We introduce SHIFT-LLM, a training-free post-pruning correction framework that inserts a Linear Residual Adapter (LRA) at each pruning site. Each LRA preserves the identity pathway of the original residual block and adds a lightweight affine residual correction. This correction is calibrated via closed-form least-squares regression on a small held-out set, without gradient computat

Read original source ↗ Open in workspace

recordType

paper

region

Global