--- title: "vLLM KV cache manual memory drift" description: "Reuse a manual vLLM KV-cache byte budget safely by measuring startup free-memory drift per GPU, preserving headroom, and gating rollouts." canonical_url: "https://fanout.sh/blog/vllm-kv-cache-manual-memory-drift" md_url: "https://fanout.sh/blog/vllm-kv-cache-manual-memory-drift.md" last_updated: "2026-10-22" access: "public" --- # vLLM KV cache manual memory drift Reuse a manual vLLM KV-cache byte budget safely by measuring startup free-memory drift per GPU, preserving headroom, and gating rollouts. - Author: Suraj Gaud - Published: 2026-10-22 - Track: Inference engineering - Access: Public - Tags: vLLM, KV cache, GPU memory, memory profiling, CUDA graphs, capacity planning vLLM KV cache manual memory drift is the gap between the GPU state that produced a safe byte reservation and the state present on a later boot. A copied value can be too large after a co-tenant, runtime, graph shape, or model configuration changes. It can also be too small and give away concurrency when more memory is free. Treat the manual value as a profiled artifact with a compatibility envelope, not as a permanent property of the GPU model. ## vLLM KV cache manual memory drift vLLM normally profiles model execution to estimate how much GPU memory can become KV cache. Its[optimization guide](https://github.com/vllm-project/vllm/blob/main/docs/configuration/optimization.md)describes a faster path. On a profiled boot, vLLM logs a value for `--kv-cache-memory`. Passing that exact byte count on a later boot skips the memory measurement and CUDA-graph memory estimation used to size the cache. The shortcut has a strict boundary. The guide says the value is valid on the same GPU with the same initial free memory. A hardware or co-tenant change requires a new profile when the old value causes an allocation failure. The[cache configuration](https://github.com/vllm-project/vllm/blob/main/vllm/config/cache.py)stores the Python-side value as `kv_cache_memory_bytes`. When set, it overrides `gpu_memory_utilization` for cache sizing. This is manual control, not another utilization hint. A deployment that sets both should record that the byte value wins. ## What the shortcut still does Skipping memory profiling does not mean skipping all startup work. The current[GPU worker implementation](https://github.com/vllm-project/vllm/blob/main/vllm/v1/worker/gpu_worker.py)still runs a profile pass used to compile the model for the configured token shape. It then reserves the supplied KV bytes and logs the current initial free memory. The same log tells operators to compare that number with the boot where the value was suggested. Without the override, the worker measures non-KV memory and transient activation headroom. It also estimates CUDA-graph memory when graph capture is enabled, then subtracts those terms from the requested memory budget. The manual branch bypasses that sizing calculation. It does not prove that weights, workspaces, graph pools, communication buffers, and other allocations still leave the same room. Fanout's[KV-cache memory formula](/blog/kv-cache-formula-llm-inference-memory)estimates the logical cache payload. The startup profile answers a different question: how many of those bytes fit beside this runtime on this boot. ## Carry the original headroom forward Record four values from the reference boot for every GPU rank: - Initial free memory before the cache allocation - Suggested manual KV-cache bytes - Model, runtime, driver, and parallel configuration - The workload shape used by startup compilation and graph capture Call reference free memory Fref and the accepted manual cache reservation Bref. On a later boot, call initial free memory Fboot. A conservative adjusted value is: Badjusted = Bref - max of zero and Fref minus Fboot - G G is an extra guard for measurement noise and unmodeled allocations. This rule pays for a loss in startup free memory before it spends any bytes on cache. Suppose the reference boot had 42 GiB free and accepted a 30 GiB cache. A canary later starts with 40.75 GiB free. The observed drift is 42 minus 40.75, or 1.25 GiB. With a 0.5 GiB guard, the adjusted cache is 30 minus 1.25 minus 0.5, or 28.25 GiB. Do not automatically add bytes when Fboot exceeds Fref. More free memory can be temporary. Re-profile before raising the reservation so graph and activation terms are measured again. ## Use the weakest GPU rank The cache setting is per GPU, while a distributed engine succeeds only when every participating rank can initialize its allocation. If four tensor-parallel ranks report 41.8, 42.1, 40.6, and 41.9 GiB free, the 40.6 GiB rank defines the immediate safety bound. An average of 41.6 GiB would hide that weak rank. The same mistake appears when a deployment copies one host's logged recommendation across a fleet with different display services, monitoring agents, or colocated jobs. Compute Badjusted for each rank, then use the minimum compatible result for a uniform launch configuration. If that minimum sacrifices too much capacity, split the fleet into verified hardware and runtime classes. Do not let one global value assume identical startup state that the logs disprove. Fanout's[paged KV-cache block-size guide](/blog/paged-kv-cache-block-size)explains how a byte budget becomes an integer block pool. A small byte reduction can lose a whole set of blocks after layout and block-size rounding. Report both bytes and resulting cache tokens. The operational effect is the change in concurrency, not the prettiness of the byte number. ## Configuration drift is not one number Initial free memory catches co-tenant and host differences, but it cannot certify configuration compatibility by itself. A model revision can change weight or workspace memory. Tensor parallelism changes the per-rank weight layout. Token and sequence limits can change compilation shapes and graph captures. KV dtype changes bytes per cached token. Multimodal settings can add processor or encoder allocations. Sleep mode introduces a managed allocation path with different lifetime rules. A public[vLLM sleep-mode issue](https://github.com/vllm-project/vllm/issues/40256)reported 5.16 GiB as available cache memory with sleep mode and 3.17 GiB without it on the described setup. That 1.99 GiB discrepancy was a bug report, not a portable correction factor. It shows why an apparently precise cache recommendation must stay tied to the allocator and runtime version that produced it. Build a compatibility key from the exact model revision, vLLM commit or release, GPU type, driver, KV dtype, parallel layout, graph mode, token limits, and relevant allocator features. Any key change sends the deployment through a fresh profiled canary. Initial free-memory comparison is for repeated boots inside one compatible class. ## Turn drift into a rollout gate Keep the first canary on automatic profiling. Capture its suggested byte value, initial free memory, cache token capacity, and successful warmup state. Start a second canary with the proposed manual value. Require its initial free memory to remain above the recorded floor plus the chosen guard. Then run the production-shaped warmup and a short load test. Watch allocation failures, CUDA out-of-memory events, cache-block count, preemptions, and request latency. A successful allocation is necessary but insufficient. A very conservative value can boot cleanly while shrinking the block pool enough to increase recomputation or reject the intended concurrency. Compare the manual canary with the profiled canary under the same request mix. The expected benefit is startup time. The acceptable cost is a measured and bounded capacity difference. Roll out by host class, not to the whole fleet at once. Stop when any rank crosses the free-memory floor or when cache tokens fall below the admission plan. ## Know when manual bytes are the wrong tool Manual bytes fit controlled images where startup state is repeatable and faster boots matter. They fit poorly on shared GPUs whose free memory changes without a deployment event. They also fit poorly when autoscaling regularly mixes GPU types, runtime versions, or model configurations under one launch template. In those environments, automatic profiling spends startup time to buy current evidence. That trade can be cheaper than failed boots, emergency cache cuts, or silent concurrency loss. Keep a re-profile path even in a stable fleet. A manual value without a current comparison boot becomes an undocumented memory promise. The safe operating rule is narrow: reuse bytes only inside the profiled compatibility class, subtract observed free-memory loss, preserve a guard, and let the weakest rank set the launch value. --- This representation contains public Fanout content only. Protected Pro lessons, account data, billing, checkout, and pricing are not included. Browse the public content map: https://fanout.sh/sitemap.md