Moving Averages (EMA)

Smooth a noisy loss with simple and exponential moving averages, see how β sets the response, and when a zero-started EMA needs 1 − βᵗ bias correction.

Imagine watching a model's loss values arrive: 8, 4, 6, 2. The latest value moves around, making the underlying trend hard to see. A moving average smooths the sequence.

One option is to average the last three values. After 8, 4, 6, the average is 6. When 2 arrives, drop 8 and average 4, 6, 2 to get 4. This is a simple moving average with a window of three.

Blend the new value with the old estimate

An exponential moving average , or EMA, keeps a single running estimate. For this example, initialize it with the first value, 8. Each later estimate is half the old estimate plus half the new observation:

New observation Calculation New estimate --- --- --- 8 Initialize with the first observation 8 4 0.5 × 8 + 0.5 × 4 6 6 0.5 × 6 + 0.5 × 6 6 2 0.5 × 6 + 0.5 × 2 4

Predict: If the next observation is 10, what is the new estimate? Does the old estimate disappear immediately?

answer It becomes 0.5 × 4 + 0.5 × 10 = 7. Half of the previous estimate remains. A single high observation pulls the estimate upward without replacing it entirely.

The three-value simple moving average would instead use 6, 2, 10 and give 6. The two methods happened to agree before this update; they use different weighting rules.

The general rule uses β for the fraction of the old estimate we retain:

xₜ is the new observation, vₜ₋₁ the previous estimate, and vₜ the updated estimate. The subscript t counts time steps. At β = 0.9, each update keeps 90% of the previous estimate and mixes in 10% of the new value. It reacts more slowly than β = 0.5.

Change the amount of smoothing

This ordinary Python example uses the same four observations: