SOURCE-LINKED INTELLIGENCE

Too much of a good thing -- when knowledge distillation promotes overfitting, and how to avoid it

arXiv · AI, language, vision and robotics · article · Aug 24, 2026 · UTC

The growing size of Convolutional Neural Networks has led to increasingly large and costly models. Knowledge Distillation (KD) addresses this by transferring knowledge from a large network (teacher) to a small one (student), also reducing the training data required. KD is traditionally applied only at the network's final output. However, its behaviour when applied at intermediate network layers has received little attention. This raises the question of whether intermediate block-wise KD, which provides supervision throughout the network, could offer an advantage under specific conditions, such

Read original source ↗ Open in workspace

recordType

paper

region

Global