Optimizer-state memory mathematical notation

Optimizer-state memory is a recurring research-paper notation family. Counts auxiliary tensors such as first and second moments, master weights, and other state maintained by an optimizer.

Optimizer-state memory: Counts auxiliary tensors such as first and second moments, master weights, and other state maintained by an optimizer. Example: Training stores several arrays in addition to model weights.