Use minimum vLLM KV cache bytes per rank
For a multi-rank server, a manual vLLM KV cache byte value must fit the rank with the least safe memory. Averaging the ranks can make startup look reasonable while one GPU fails during cache allocation or graph capture.
The useful multi-rank minimum is not the smallest free-memory reading alone. Measure each rank after model loading and warmup, reserve operating headroom, then take the minimum safe KV budget across compatible ranks.
That rule leaves memory unused on stronger GPUs. With one uniform command-line value, the waste is the price of keeping every rank inside its own limit.
Why vLLM KV cache bytes use the minimum rank
vLLM defines `kv_cache_memory_bytes` as the KV-cache size per GPU. When it is set, it replaces the automatic cache budget derived from `gpu_memory_utilization`.
A distributed launch supplies one cache configuration to its workers. If every rank receives the same requested bytes, the largest safe uniform value is the smallest safe value reported by any rank.
Write the safe cache budget for rank i as B_i. The uniform budget is min(B_0, B_1, ..., B_n), rounded down to a valid cache-block allocation.
Do not sum the rank budgets for this decision. A total of 64 GiB across four GPUs does not prove that each GPU can allocate a 16 GiB local cache.
The same distinction applies to tensor, pipeline, and data-parallel layouts. Parallelism may shard model state or KV state, but a local allocation still has to fit the device that owns it.
Fanout's KV-cache memory formula explains the bytes per token. The multi-rank minimum decides how many of those bytes can be allocated safely on every worker.
Build a per-rank memory ledger
Start with a profiled launch that uses automatic sizing. vLLM's GPU worker profiler measures model and activation memory before calculating available KV-cache bytes.
Record each rank separately. A fleet-wide average hides the rank that matters.
For each rank, keep these values from the same boot:
- Initial free device memory before model loading.
- Model weights and other persistent allocations.
- Peak activation memory during the profile run.
- CUDA graph memory measured during warmup or capture.
- Memory used by other processes on that GPU.
- The cache budget vLLM allocated or suggested.
Add an operating reserve for allocator fragmentation, library drift, request-shape changes, and monitoring agents. The reserve is a policy input, not a value vLLM can infer for every deployment.
Do not combine readings from different releases. A driver, kernel, quantization, graph shape, or model configuration change can alter the non-KV footprint.
Fanout's PagedAttention guide covers how the resulting block pool is managed after allocation.
Work through a four-rank example
Assume a four-GPU server runs the same model configuration on every rank. After profiling and graph capture, the measured room for KV cache is 18.8, 17.6, 12.9, and 16.4 GiB.
Rank 2 is the ceiling. A uniform 16 GiB setting is unsafe even though the average available room is 16.4 GiB.
Now reserve 0.9 GiB on every GPU for runtime variation. The safe budgets become 17.9, 16.7, 12.0, and 15.5 GiB.
The manual setting should be no more than 12.0 GiB per GPU before block rounding. The four ranks then allocate at most 48 GiB in total.
The unused safe capacity is 5.9 GiB on rank 0, 4.7 GiB on rank 1, zero on rank 2, and 3.5 GiB on rank 3. The uniform policy strands 14.1 GiB across the stronger ranks.
That number is useful. It tells you whether fixing the weak rank is worth more than tuning the shared value.
If rank 2 is shared with another process, moving that process may recover most of the 14.1 GiB. Raising the global budget without removing the pressure only moves the failure closer.
Free memory is not the manual budget
Free device memory includes space that later allocations may need. The manual value should come from a complete non-KV ledger, not from `nvidia-smi` at an arbitrary instant.
The vLLM manual-byte configuration skips automatic memory profiling for the cache calculation.
A short model profile may still compile kernels, but the byte value itself becomes the reservation.
That makes an old value brittle. A cache budget copied from a clean boot can fail when another process starts first or a new CUDA graph consumes more memory.
A current vLLM issue shows the practical failure mode: one shared GPU capped a three-rank group, while the other two GPUs retained much more unused memory.
The report measured about 15 GiB of KV allocation per rank. The shared rank had almost no remaining headroom, while the unshared ranks each left roughly 17 GiB unused.
Those numbers do not transfer to another deployment. Preserve the per-rank ledger that produced your own budget.
Leave room for CUDA graph capture
Graph capture can happen after the cache pool is reserved. A setting that fits immediately after model loading may still fail when capture asks for another large allocation.
Measure the peak after the graph mode and request shapes used in production have warmed up. Eager-mode measurements are not a substitute if production uses captured graphs.
vLLM issue #57936 documents a separate warning: a startup suggestion can double-count CUDA graph memory and understate the usable cache budget.
An understated suggestion wastes capacity. An overstated manual value can crash. Treat the log suggestion as one ledger input, then reconcile it with observed free memory after capture.
Keep the operating reserve outside the cache value. If the weakest rank has 13.1 GiB after capture and policy requires 1 GiB free, the candidate budget is 12.1 GiB before block rounding.
Repeat the warmup several times. Use the worst stable observation, not the median, when the goal is a uniform setting that must boot every rank.
Convert bytes into serving capacity
A smaller weakest-rank budget reduces the block count and the maximum resident token capacity. Translate that reduction into a concurrency limit before accepting it.
Suppose the model's local KV cost is 256 KiB per token per rank. A 12 GiB cache holds about 49,152 token slots before allocator and block-rounding effects.
At an 8,192-token maximum sequence length, that is roughly six full-length sequences per rank. Real batches can serve more requests when their active contexts are shorter.
Do not divide by the global KV bytes per token if the cache is sharded. Use the local bytes owned by one rank under the actual TP, PP, DP, DCP, or PCP layout.
Compare the calculated capacity with vLLM's startup concurrency estimate and a load test. A mismatch can reveal a wrong layer count, cache dtype, block boundary, or parallelism assumption.
Choose enough cache for the target concurrency while keeping the weakest rank outside its failure envelope.
Decide whether uniform sizing is acceptable
Uniform sizing is reasonable when ranks use identical GPUs, have stable co-tenancy, and show a small spread in safe memory.
It becomes expensive when one rank is shared, has a smaller memory slice, loads extra components, or repeatedly reports a different graph footprint.
Track the minimum, median, and maximum safe budget. The ratio between minimum and median shows how much one rank constrains the group.
If the minimum is 12 GiB and the median is 18 GiB, the weak rank removes one-third of the typical cache capacity. That is an infrastructure problem before it is a cache-tuning problem.
Possible remedies include isolating the noisy process, making GPU slices homogeneous, or separating unlike workers into different serving groups.
Do not assign a different manual value to ranks unless the vLLM version and topology explicitly support asymmetric pools. Shared scheduling assumptions may require equal block counts.
Revalidate every manual value
Store the manual byte setting beside its evidence: vLLM commit, model revision, quantization, cache dtype, parallel layout, driver, GPU type, graph mode, and per-rank ledger.
Reprofile after any of those inputs changes. Also reprofile when startup free memory shifts, even if the application code did not.
At boot, fail the deployment if a rank's observed post-warmup reserve falls below policy. Do not wait for a production prefill to discover the gap.
During load tests, watch OOMs, preemptions, cache occupancy, active tokens, and graph-capture allocations. Confirm that the chosen reserve remains present at the highest supported request shape.
Calculate every rank, subtract measured headroom, round down, and deploy the minimum. Report the stranded capacity so the team can decide whether to fix the weak rank.