Pipeline Parallelism

Between layer 15 and layer 16 of the course model, one pair of tensors leaves and nothing else does. Give that boundary to a cable and the model runs on two devices, each holding whole layers, with far fewer bytes crossing per step than the within-layer division in the tensor parallelism lesson needed. What the cable costs is order: the…

In this lesson

• The partition rule
• What each stage holds
• What crosses the boundary
• One request through two stages
• Filling the pipe

Continue learning

Pro unlocks the complete lesson, video, practice, and code where available.