Gradient-descent parameter update

Updates parameters by taking a learning-rate-sized step opposite the local loss gradient.

Theta at time t plus one equals theta at time t minus eta times the gradient with respect to theta of the loss at theta t.

Measure which way makes loss rise fastest, then move a controlled distance the other way.

The equals sign defines an iterative update, while subtraction reverses the gradient's uphill direction.

This is an update rule rather than the optimization objective itself; optimizers such as momentum and Adam modify the direction or scaling.