KV cache reserve under tensor parallelism
KV cache reserve under tensor parallelism is a per-rank budget, not a cluster-wide pile of free VRAM. Each request needs a KV shard on every tensor-parallel rank, so the tightest rank sets token capacity.
The easy mistake is to subtract weights from total fleet memory, then treat every leftover byte as KV cache. That counts memory the runtime already needs for activations, CUDA graphs, communication, and safety margin.
KV cache reserve under tensor parallelism
vLLM profiles a forward pass before it fixes the KV pool.
Its GPU worker implementation subtracts non-KV memory and CUDA-graph estimates from the requested memory on each rank.
That gives a useful per-rank equation: KV bytes equal requested memory minus weight shard, persistent runtime memory, profiled activation peak, and graph memory.
Requested memory is total device memory times gpu_memory_utilization. The unused share outside that target is headroom. It is not the same pool as the activation reserve inside the target.
The Elastic KV Cache paper makes this distinction explicit. Raising utilization captures stranded headroom, while changing the prefill chunk changes the profiled activation reserve.
Keep those two rows separate in the ledger. Otherwise one adjustment can appear to free the same bytes twice.
Build the ledger per rank
Start with the startup log from every rank, not one representative GPU. Mixed free memory, another process, or an uneven workspace can make one rank smaller than its peers.
For each rank, record:
- total device memory
- initial free memory at startup
- requested memory after gpu_memory_utilization
- model weight shard
- persistent runtime and communication buffers
- CUDA-graph memory
- peak activation memory
- available KV-cache memory
Suppose each GPU has 80 GiB and utilization is 0.90. The requested region is 72 GiB, leaving 8 GiB outside the runtime budget.
Now use an illustrative rank with 18 GiB of weights, 4 GiB of persistent buffers, 3 GiB of CUDA graphs, and 3 GiB of peak activations.
The KV pool is 72 minus 18 minus 4 minus 3 minus 3, which equals 44 GiB on that rank. The separate 8 GiB headroom remains outside the equation.
Do not write the budget as 80 minus those four rows. That would silently consume the 8 GiB margin selected by the utilization setting.
Fanout's KV-cache memory formula explains how model shape and cache dtype determine bytes per token. This ledger determines how many bytes are available to spend.
Tensor parallelism does not multiply token capacity
Tensor parallelism usually shards model weights and attention heads. That reduces both weight bytes and KV bytes per token on each rank.
The cluster still stores one logical token across all ranks. If one token needs a 128 KiB KV shard per rank, a 44 GiB pool holds 360,448 token positions per rank.
Four ranks do not turn that into 1,441,792 token positions. Every logical position consumes one shard on each of the four GPUs, so the request stops when any rank reaches 360,448.
Aggregate KV bytes help compare hardware cost. Per-rank token capacity decides whether the scheduler can admit another sequence.
This is why a single rank with 42 GiB of usable KV and three ranks with 44 GiB behave like a 42 GiB pool for admission. The extra 2 GiB on the other ranks cannot hold complete distributed tokens alone.
Use the maximum measured bytes per token across ranks when layouts are uneven. Optimistic averaging converts skew into an out-of-memory event.
The activation reserve shrinks with TP
The activation reserve does not keep a fixed fraction of the KV pool as TP rises. Some tensors shard, while some graph, communication, and runtime costs stay per device.
In the Elastic KV Cache measurements, the reserve was 16% of KV for a 7B model at TP1. It fell to 2.7% for that model at TP4 and measured 7.7% for a 32B model at TP4.
Those are results for one A100-40GB setup, Qwen2.5 models, vLLM 0.23.0, and the paper's workload. They are evidence of the direction, not constants to paste into another deployment.
The practical consequence is slightly awkward. TP frees weight memory, but it can also make activation-reserve reclamation a smaller optimization than the fleet-level diagram suggests.
Measure the reserve on each target TP degree. Do not divide a TP1 activation peak by four and assume the rest of the memory plan scales cleanly.
Communication workspaces and CUDA graphs can change at the same time. The current vLLM profiler records these costs before reporting available KV bytes.
Compare chunking before reclaiming memory
A large max_num_batched_tokens value raises the largest prefill step the profiler must support. That can increase the activation peak and reduce the fixed KV pool.
The Elastic KV Cache paper built a working allocator that lends this reserve to KV during decode and returns it before prefill. A static 3 GiB reclamation crashed on a large prefill burst.
The dynamic mechanism worked, but the paper found a simpler result. Reducing the prefill chunk from 32,768 to 8,192 tokens raised median TTFT by about 1% while increasing KV capacity from 314K to 375K tokens.
Its elastic mode reached about 364K tokens at a similar median TTFT. On that workload, smaller chunked prefill recovered more KV than dynamic reserve lending.
Fanout's prefill and decode guide explains why prefill is compute-heavy while each decode step advances active sequences by roughly one token.
Run the same comparison under your prompt lengths and concurrency. A result from one 7B model does not settle a large MoE deployment or a different attention backend.
Convert bytes into an admission limit
KV capacity becomes useful only after it is translated into the scheduler's units.
If a rank has 44 GiB for KV and each token consumes 128 KiB on that rank, raw capacity is 360,448 token positions.
Do not advertise all of them. Reserve blocks for in-flight growth, allocator granularity, speculative lookahead, and any cache groups whose usable boundaries differ.
Suppose policy keeps 8% free for growth. The schedulable limit becomes floor of 360,448 times 0.92, or 331,612 token positions.
If the serving mix averages 12,000 live token positions per request, the arithmetic ceiling is 27 concurrent requests. That is a memory ceiling, not a throughput promise.
The PagedAttention guide covers why blocks reduce fragmentation but do not remove this capacity limit.
Use measured block bytes when possible. Dtype scales, hybrid cache groups, alignment, and backend metadata can move the result away from a textbook model formula.
Audit the budget after every material change
Record the ledger beside benchmark results for every model, TP degree, cache dtype, graph mode, and prefill chunk.
After an upgrade, compare available KV bytes and each subtracted row. A smaller pool is not automatically a leak if the new build profiles graph or workspace memory more completely.
Watch the minimum rank, not the average. Alert when initial free memory differs across ranks before model loading because that difference will usually survive into the KV budget.
Load-test the largest allowed prefill while the KV pool is near its admission limit. This verifies that the activation row is real protection rather than unused padding.
Then lower max_num_batched_tokens and repeat. If TTFT barely moves while KV capacity rises, chunking is the cheaper lever.
KV cache reserve under tensor parallelism is easiest to reason about as a set of per-rank ledgers. Separate outside headroom from inside reserves, then let the smallest measured rank set the token limit.