Pipeline parallelism mathematical notation

Pipeline parallelism is a recurring research-paper notation family. Assigns consecutive model layers to different stages and streams microbatches through them to share model memory across devices.

Pipeline parallelism: Assigns consecutive model layers to different stages and streams microbatches through them to share model memory across devices. Example: Run one group of layers on each stage in sequence.