Learning Rate, Decay

Neural Network from Scratch

A glimpse of Learning Rate, Decay

A gradient describes how loss changes near the current parameter values. A gradient update uses that information to change the parameters; its learning rate sets the step size. A schedule changes that rate over a training run. Learning-rate decay makes it smaller over time, often to allow smaller adjustments later.

Smaller later steps are not automatically better. We can see the trade-off on a loss whose best answer is already known.

Continue learning

Pro unlocks guided learning, complete notes, practical walkthroughs, exercises, and every AI Research lesson.