---
title: "Adam Optimizer"
description: "How Adam keeps two running averages per parameter, one of gradients and one of their squares. Compute the first update and its bias correction by hand."
canonical_url: "https://fanout.sh/ai/lessons/neural-network-from-scratch-adam-optimizer"
md_url: "https://fanout.sh/ai/lessons/neural-network-from-scratch-adam-optimizer.md"
access: "pro-preview"
---

# Adam Optimizer

How Adam keeps two running averages per parameter, one of gradients and one of their squares. Compute the first update and its bias correction by hand.

- Access: Fanout Pro (public preview)
- Module: Neural Network from Scratch

## A glimpse of Adam Optimizer

A model's loss can respond strongly to a small change in one parameter and weakly to a change in another. Those local rates form its gradient. Ordinary gradient descent multiplies each gradient entry by the same learning rate. Adam also keeps a short history for each parameter and uses that history to scale its updates.

There are two stored averages: one of the gradients, and one of their squares. The first tracks direction over time. The second tracks magnitude without canceling positive and negative gradients.

## Continue learning

Pro unlocks guided learning, complete notes, practical walkthroughs, exercises, and every AI Research lesson.

---
This representation contains public Fanout content only. Protected Pro lessons, account data, billing, checkout, and pricing are not included.

Browse the public content map: https://fanout.sh/sitemap.md