---
title: "Learning Rate, Decay"
description: "Compare a constant and a decaying learning rate on a loss whose minimum is known, then describe a schedule so it means one thing, warm-up included."
canonical_url: "https://fanout.sh/ai/lessons/neural-network-from-scratch-learning-rate-decay"
md_url: "https://fanout.sh/ai/lessons/neural-network-from-scratch-learning-rate-decay.md"
access: "pro-preview"
---

# Learning Rate, Decay

Compare a constant and a decaying learning rate on a loss whose minimum is known, then describe a schedule so it means one thing, warm-up included.

- Access: Fanout Pro (public preview)
- Module: Neural Network from Scratch

## A glimpse of Learning Rate, Decay

A gradient describes how loss changes near the current parameter values. A gradient update uses that information to change the parameters; its learning rate sets the step size. A schedule changes that rate over a training run. Learning-rate decay makes it smaller over time, often to allow smaller adjustments later.

Smaller later steps are not automatically better. We can see the trade-off on a loss whose best answer is already known.

## Continue learning

Pro unlocks guided learning, complete notes, practical walkthroughs, exercises, and every AI Research lesson.

---
This representation contains public Fanout content only. Protected Pro lessons, account data, billing, checkout, and pricing are not included.

Browse the public content map: https://fanout.sh/sitemap.md