Kullback-Leibler divergence

Measures the expected log-density mismatch when outcomes follow p but are represented using q.

K L divergence from p to q equals the sum over x of p of x times the logarithm of p of x divided by q of x.

Ask how surprised you would be, on average under p, if you used q's probabilities instead.

The p-weighted summation is an expectation: average the log probability ratio under outcomes drawn from p.

KL is a divergence rather than a metric: reversing p and q generally changes the value, and support mismatch can make it infinite.