Data parallelism mathematical notation

Data parallelism is a recurring research-paper notation family. Replicates model computation across ranks that process different examples, then synchronizes parameter gradients or updates.

Data parallelism: Replicates model computation across ranks that process different examples, then synchronizes parameter gradients or updates. Example: Each of p workers processes a different local portion of the batch.