---
title: "Distilling the Knowledge in a Neural Network"
description: "Knowledge distillation turns an expensive ensemble or large teacher into a smaller, deployable student. Read it to see why the full pattern of class probabilities carries more useful structure than hard labels alone, and how temperature-scaled soft targets transfer that structure during training."
canonical_url: "https://fanout.sh/daily/2026-07-29-knowledge-distillation"
md_url: "https://fanout.sh/daily/2026-07-29-knowledge-distillation.md"
last_updated: "2026-07-29"
access: "public"
---

# Distilling the Knowledge in a Neural Network

Knowledge distillation turns an expensive ensemble or large teacher into a smaller, deployable student. Read it to see why the full pattern of class probabilities carries more useful structure than hard labels alone, and how temperature-scaled soft targets transfer that structure during training.

## Paper details

- Authors: Hinton, Vinyals, and Dean

- Venue: NIPS 2014 Deep Learning Workshop

- Track: AI Research

- Difficulty: Easy

- Reading time: 30 min

- Original paper: https://arxiv.org/abs/1503.02531

## What you will learn

- How a teacher's soft class probabilities reveal similarities between classes that one-hot labels discard.

- Why raising the softmax temperature exposes more of that relative probability structure for the student to learn.

- How combining soft teacher targets with the original hard labels produces a compact model that is easier to deploy.

## Continue exploring

[Explore Knowledge Distillation](https://fanout.sh/knowledge-graph?node=ai-research%3Aconcept%3Aknowledge-distillation): Open Fanout's knowledge-distillation node and connect the paper to teacher-student training, soft targets, temperature scaling, and model compression.

---
This representation contains public Fanout content only. Protected Pro lessons, account data, billing, checkout, and pricing are not included.

Browse the public content map: https://fanout.sh/sitemap.md