--- title: "Size vLLM KV cache bytes after CUDA graphs" description: "Turn vLLM's manual KV-cache suggestion into a repeatable byte budget using measured CUDA-graph slack, an operating reserve, and block rounding." canonical_url: "https://fanout.sh/blog/vllm-kv-cache-bytes-cuda-graph-headroom" md_url: "https://fanout.sh/blog/vllm-kv-cache-bytes-cuda-graph-headroom.md" last_updated: "2026-10-24" access: "public" --- # Size vLLM KV cache bytes after CUDA graphs Turn vLLM's manual KV-cache suggestion into a repeatable byte budget using measured CUDA-graph slack, an operating reserve, and block rounding. - Author: Suraj Gaud - Published: 2026-10-24 - Track: Inference engineering - Access: Public - Tags: vLLM, KV cache, CUDA graphs, GPU memory, capacity planning, inference operations Sizing vLLM KV cache bytes after CUDA graph capture is safer than copying one startup suggestion. The manual value skips the measurements that produced it, so the next boot must inherit enough room for capture and runtime variation. The useful number is the cache already allocated plus repeatable post-capture slack, minus a reserve. Measure that quantity across several cold boots, round it down to cache blocks, and keep the smallest result. A larger byte value may boot once and still fail after a driver change, a different graph shape, or another process claims memory first. ## Why vLLM KV cache bytes need graph headroom vLLM's[optimization guide](https://docs.vllm.ai/en/latest/configuration/optimization/)says startup logs the exact manual cache value that reproduces the current allocation. Passing `--kv-cache-memory` on the next boot skips memory profiling and the CUDA-graph memory estimation pass. vLLM reserves the byte count directly instead of deriving it from `gpu_memory_utilization`. That makes the logged value a snapshot, not a permanent property of the model. The documentation limits it to the same GPU with the same initial free memory. CUDA graphs occupy memory outside the KV block pool. So do model weights, allocator state, non-Torch allocations, temporary activations, communication buffers, and any co-tenant process. Manual sizing has to leave all of those allocations room to repeat. A value that consumes every free byte before graph capture is not a capacity plan. Fanout's[KV-cache memory formula](/blog/kv-cache-formula-llm-inference-memory)gives the model's bytes per retained token. This post answers the other half: how many bytes the GPU can safely give that cache. ## Record three values from each boot For each cold boot, record the cache allocation, free memory after the production graph set finishes capturing, and the operating reserve required by policy. Use one configuration for the whole series. Model revision, quantization, cache dtype, tensor parallel size, speculative decoder, maximum sequence length, and graph mode all belong in the run record. Call the allocated cache C. Call post-capture free memory F. Call the required reserve R. The full-device candidate for that boot is C plus F minus R. If F is smaller than R, do not increase the cache. Reduce it by at least R minus F. Repeat the boot and use the smallest candidate. A median hides the exact low-memory start that a fixed manual value must survive. Keep the logged vLLM suggestion beside these values. It is useful evidence, but it should agree with the observed ledger before it becomes a deployment input. Also save the final free memory after a short load test. Some buffers appear only when request shapes reach the production maximum. ## Work through a repeated-boot example Assume automatic sizing allocates an 18.0 GiB KV cache. After graph capture and warmup, five cold boots leave 1.6, 1.2, 1.4, 1.1, and 1.3 GiB free. The deployment requires 0.8 GiB of operating reserve. The lowest observed slack is 1.1 GiB. The candidate is 18.0 plus 1.1 minus 0.8, or 18.3 GiB. The calculation recovers 0.3 GiB while preserving the measured reserve on the weakest boot. Do not set 18.8 GiB because the best boot left 1.6 GiB free. That choice spends slack that did not appear on four of the five starts. Do not set 19.1 GiB merely because cache plus minimum free memory equals 19.1 GiB. That leaves no room for the policy reserve. If the existing cache is not identical across boots, calculate C plus F minus R for every run and take the minimum of those results. The example assumes one GPU. For multiple ranks, calculate each rank separately and then use the smallest safe local value when the launch requires one uniform setting. ## Check whether the startup suggestion is credible A reported value can be wrong in either direction. Treat issue reports as version-specific evidence, then compare their failure mode with the release you run. One vLLM issue reports that the[`--kv-cache-memory` suggestion double-counted CUDA-graph memory](https://github.com/vllm-project/vllm/issues/57936). In that report, the live allocation was correct but the suggested fit value became too small or even negative. Copying it would waste capacity or disable the startup-plan path. A separate open report describes[FP8 KV cache startup failing after an underestimated graph allocation](https://github.com/vllm-project/vllm/issues/60350). That case allocated about 19.93 GiB to KV cache on a 24 GiB GPU, then used about 0.80 GiB during graph capture. The nearly zero graph estimate left too little room and startup ran out of memory. These reports point in opposite directions. One suggestion subtracts graph memory twice. The other estimate reserves too little for the graph that is actually captured. The response is the same: identify the exact vLLM commit, inspect the memory ledger, and verify it with measured post-capture slack. ## Capture the graph shapes production uses Graph memory depends on the shapes and modes vLLM captures. A short eager run does not establish headroom for a deployment that serves with CUDA graphs. Warm the model with the graph configuration used in production. Include the largest supported batch and token shapes that the capture policy can select. Speculative decoding, multimodal inputs, and hybrid architectures can change the captured set. Treat enabling one of them as a new sizing experiment. The[vLLM GPU worker](https://docs.vllm.ai/en/latest/api/vllm/v1/worker/gpu_worker/)profiles model and activation memory before deciding how much remains for KV cache. A manual byte value bypasses that decision on later boots. Keep a control boot without the flag so the current profiler can expose drift. Run the same cold-start sequence after changing vLLM, PyTorch, CUDA, the driver, the attention backend, or the graph compilation settings. If startup timing matters, compare the saved manual path with the profiled control. Faster startup is useful only when the byte value still passes the full warmup and load test. ## Round the byte result to usable blocks The cache manager allocates blocks, not fractional tokens. Convert the safe byte result into whole blocks before writing the flag. Suppose local KV state costs 256 KiB per token and the cache block holds 16 tokens. One block then costs 4 MiB. An 18.3 GiB candidate contains 18,739.2 MiB. Dividing by 4 MiB gives 4,684.8 blocks, so round down to 4,684 blocks. That is 74,944 token slots and 18,736 MiB of cache. The remainder stays unused because it cannot form another complete block. Compare that count with the startup concurrency estimate and a load test. Fanout's[paged KV-cache block-size guide](/blog/paged-kv-cache-block-size)explains how block geometry also affects tail waste. Use local KV bytes per token under the actual parallel layout. Dividing by a global, unsharded model total can understate the number of tokens that one rank can hold. ## Validate reserve under load A successful boot proves only that initialization fit. The reserve must survive production request shapes and the allocator's steady-state behavior. Replay a representative mix of prompt lengths, output lengths, and concurrency. Include the largest supported request and a sustained run near the intended saturation point. Record minimum free memory, cache occupancy, preemptions, recomputations, request failures, and latency percentiles. If the reserve disappears while cache occupancy remains low, the missing memory is not a KV-capacity problem. Inspect graph selection, temporary activations, communication, and non-Torch allocations. If occupancy reaches the limit and preemptions rise while reserve remains intact, the byte setting is safe but too small for the target workload. Raise it only from a new measured ledger. Do not spend the reserve because a throughput graph looks flat in one run. ## Reprofile instead of carrying the number forward Store the manual byte value with its evidence: hardware, initial free memory, model, cache dtype, graph settings, software revisions, repeated-boot observations, reserve, and block rounding. Set an invalidation rule for every field. A changed input means the old byte value becomes a hypothesis and automatic profiling becomes the control again. Also invalidate the value when cold-boot free memory moves outside the measured range. That catches new agents, display processes, or co-tenants without requiring an application release. A manual cache value is safe when it reproduces a measured allocation and preserves headroom after the actual graph set is captured. Use the smallest repeated-boot result, subtract an explicit reserve, round down to blocks, and confirm the reserve under load. --- This representation contains public Fanout content only. Protected Pro lessons, account data, billing, checkout, and pricing are not included. Browse the public content map: https://fanout.sh/sitemap.md