All-to-all collective mathematical notation

All-to-all collective is a recurring research-paper notation family. Redistributes distinct tensor chunks from every rank to every other rank, commonly for sparse expert dispatch and return.

All-to-all collective: Redistributes distinct tensor chunks from every rank to every other rank, commonly for sparse expert dispatch and return. Example: Every device sends a different payload to every destination.