Tensor Parallelism

A large layer can be divided so that several GPUs work on the same tokens at the same time. Each holds a different slice of the layer's weights. The useful question is where those slices can compute independently and where their results must meet before the model can continue.

In this lesson

• Output slices and partial sums
• A two-worker calculation
• Llama's feed-forward block
• Attention heads and partition limits
• The next operation's dependency

Continue learning

Pro unlocks the complete lesson, video, practice, and code where available.