Row-parallel tensor communication

Shows why sharding a linear layer across its input dimension creates full-shaped partial outputs that require a sum collective.

Each rank r computes local output Y r as input shard X r times weight shard W r, then global Y is the sum all-reduce of all rank-local partial outputs from rank one through p.

Every device computes part of each output activation, then the devices add those parts together.

Sum all-reduce is the distributed reduction that reconstructs the mathematical result of the unsharded matrix product.

Column-parallel layers instead produce distinct output shards; Megatron pairs row and column partitions to avoid unnecessary collectives between adjacent operations.