Context Parallelism

For a long enough request, the model's weights stop being the largest thing on the device. One request at the course model's full context holds 17.2 GB of keys and values against 16.06 GB of weights, the comparison the KV capacity lesson made. Every layout in this module so far leaves that cache whole on one device or divides it…

In this lesson

• Where head-parallel division stops
• Attention over a shard of positions
• The prefill form and the decode form
• What the pin can and cannot run
• Practice: shard and merge

Continue learning

Pro unlocks the complete lesson, video, practice, and code where available.