SOURCE-LINKED INTELLIGENCE

Six Layers Less: Encoder Pruning for Whisper with Label-Free Recovery

arXiv · AI, language, vision and robotics · article · Sep 23, 2026 · UTC

Pruning large pre-trained transformer-based ASR models such as OpenAI's Whisper has seen great adoption, as pruning the decoder led to significant end-to-end transcription speedups. For instance, the {\tt whisper-large-v3-turbo} variant reduced the decoder from 32 to 4 layers, while Distill-Whisper similarly reduced the decoder to only 2 layers. Although some attention has been put towards reducing the size of the encoder, no approach has seen wide adoption. This could be due to the need for custom inference implementations to take advantage of the compressed model. We present an approach that

Read original source ↗ Open in workspace

recordType

paper

region

Global