Entropy & Information Theory

Math Fundamentals

A glimpse of Entropy & Information Theory

A coin that always lands heads is easy to predict. A fair coin leaves you unsure. Entropy puts a number on that uncertainty, using the probabilities you assign to the outcomes.

Start with a simpler question: how surprising is one outcome? An outcome with probability 1/2 carries one bit of surprise. Probability 1/4 carries two bits; probability 1/8 carries three. Each halving adds a bit.

Continue learning

Pro unlocks guided learning, complete notes, practical walkthroughs, exercises, and every AI Research lesson.