KL Divergence

Math Fundamentals

A glimpse of KL Divergence

Suppose an outcome is red 75% of the time and blue 25% of the time. Call that distribution P = [0.75, 0.25]. A model predicts Q = [0.5, 0.5]. How much does using Q cost us, compared with predicting the actual probabilities?

We need a scoring rule. Give a prediction a cost of −log₂(q) when an outcome assigned probability q happens. Predicting probability 1/2 costs one bit; predicting probability 1/4 costs two bits.

Continue learning

Pro unlocks guided learning, complete notes, practical walkthroughs, exercises, and every AI Research lesson.