---
title: "Context Parallelism"
description: "When one long request's KV cache outgrows the weights: attention over a shard of positions, the prefill and decode forms, and what an engine can run."
canonical_url: "https://fanout.sh/inference-eng/curriculum/18-08"
md_url: "https://fanout.sh/inference-eng/curriculum/18-08.md"
access: "pro-preview"
---

# Context Parallelism

When one long request's KV cache outgrows the weights: attention over a shard of positions, the prefill and decode forms, and what an engine can run.

- Access: Fanout Pro (public preview)

## Public preview

For a long enough request, the model's weights stop being the largest thing on the device. One request at the course model's full context holds 17.2 GB of keys and values against 16.06 GB of weights, the comparison the KV capacity lesson made. Every layout in this module so far leaves that cache whole on one device or divides it…

## In this lesson

- Where head-parallel division stops
- Attention over a shard of positions
- The prefill form and the decode form
- What the pin can and cannot run
- Practice: shard and merge

## Continue learning

Pro unlocks the complete lesson, video, practice, and code where available.

---
This representation contains public Fanout content only. Protected Pro lessons, account data, billing, checkout, and pricing are not included.

Browse the public content map: https://fanout.sh/sitemap.md