Gradients

Collect two partial derivatives into a gradient, take one gradient descent step by hand, and see why a learning rate that is too large raises the loss.

Imagine adjusting two numbers in a model, w and b. These adjustable numbers are its parameters . You want to know which small changes might lower its error score.

Use a deliberately small error function, called a loss :

The loss is smallest at w = 1, b = −2, where both squares are zero. We can see the answer here; that lets us check the method we would use when the answer is harder to see.

Ask one parameter at a time

Start at w = 3, b = −1. The loss is (3 − 1)² + (−1 + 2)² = 5.

First hold b fixed and change w. We need its derivative , the loss's local rate of change as w moves. For a square u², that rate is 2u. Here u = w − 1, which changes by the same amount as w, so the rate is 2(w − 1) = 4.

For example, increasing w from 3 to 3.01 while leaving b at −1 raises the loss from 5 to 5.0401. The change, 0.0401, is close to the rate 4 multiplied by the input change 0.01.

Next hold w fixed and change b. The same calculation gives 2(b + 2) = 2. Each is a partial derivative : we changed one input while holding the other fixed.

Collect those two partial derivatives, in the same order as the parameters:

This ordered list of rates is the gradient . An ordered list of numbers is also called a vector. The symbol ∇ means we are collecting rates of change, one per input. Compare equally small moves in different directions, using ordinary distance to measure their size. For a smooth loss, following a nonzero gradient gives the fastest local increase. At a zero gradient, it supplies no preferred direction.

To try decreasing the loss, take a small step in the opposite direction. With a step size of 0.25:

The new loss is 1² + 0.5² = 1.25. This is gradient descent . Its step size is usually called the learning rate and written η.