All-reduce collective mathematical notation

All-reduce collective is a recurring research-paper notation family. Combines equal-shaped values from every rank with a reduction and returns the identical reduced result to every rank.

All-reduce collective: Combines equal-shaped values from every rank with a reduction and returns the identical reduced result to every rank. Example: Sum or average gradients across ranks and give the result back to each worker.