Adam Optimizer

Neural Network from Scratch

A glimpse of Adam Optimizer

A model's loss can respond strongly to a small change in one parameter and weakly to a change in another. Those local rates form its gradient. Ordinary gradient descent multiplies each gradient entry by the same learning rate. Adam also keeps a short history for each parameter and uses that history to scale its updates.

There are two stored averages: one of the gradients, and one of their squares. The first tracks direction over time. The second tracks magnitude without canceling positive and negative gradients.

Continue learning

Pro unlocks guided learning, complete notes, practical walkthroughs, exercises, and every AI Research lesson.