All-gather collective mathematical notation

All-gather collective is a recurring research-paper notation family. Collects a distinct local shard from every rank and returns the concatenated global tensor to all participating ranks.

All-gather collective: Collects a distinct local shard from every rank and returns the concatenated global tensor to all participating ranks. Example: Reconstruct the complete tensor on every rank from local pieces.