Distilling the Knowledge in a Neural Network

Knowledge distillation turns an expensive ensemble or large teacher into a smaller, deployable student. Read it to see why the full pattern of class probabilities carries more useful structure than hard labels alone, and how temperature-scaled soft targets transfer that structure during training.

Reading focus: How a teacher's soft class probabilities reveal similarities between classes that one-hot labels discard. Why raising the softmax temperature exposes more of that relative probability structure for the student to learn. How combining soft teacher targets with the original hard labels produces a compact model that is easier to deploy.

NIPS 2014 Deep Learning Workshop. Hinton, Vinyals, and Dean. 30 min read, easy difficulty.