Tensor parallelism mathematical notation

Tensor parallelism is a recurring research-paper notation family. Partitions large layer tensors and their matrix operations across ranks, exchanging partial activations or reductions when required.

Tensor parallelism: Partitions large layer tensors and their matrix operations across ranks, exchanging partial activations or reductions when required. Example: Each rank stores and computes with only part of the weight matrix.