KV cache headroom for burst arrivals

KV cache headroom for burst arrivals is the memory between normal demand and the point where a traffic spike forces eviction, preemption, or rejection.

Do not choose it as a round percentage of GPU memory. Choose a reaction window, measure net KV block growth inside that window, and reserve enough blocks for a high percentile of those bursts.

The steady working set and the burst reserve answer different questions. One protects expected reuse. The other buys time for the serving system to react.

KV cache headroom for burst arrivals

The KVSET paper defines a KV cache working set as the minimum capacity needed to reach a target hit rate under a modeled eviction policy.

That curve is the baseline. It says how many reusable pages should remain after ordinary churn.

A burst reserve sits above the baseline. It covers the extra live and reusable pages created before admission control, autoscaling, offload, or a routing change can reduce pressure.

The distinction prevents double counting. A month of old cache writes does not belong in the burst reserve, and a p99 arrival spike does not belong in the steady LRU curve.

The KV-RM study shows why the distinction matters. Its paged design tracks the active set instead of retaining a worst-case arena, which leaves more multiplexing headroom under mixed-length arrivals.

Page one mostly explains KV size, compression, and tiering. It does not turn a measured burst into a block reserve with a deadline attached.

Choose the reaction window first

Headroom is useful only for a defined amount of time. Pick the longest delay before a control action can materially slow net KV growth.

That delay may include metric collection, alert evaluation, a routing update, queue draining, replica startup, and model warmup.

If admission control can act in two seconds but a new replica takes 90 seconds, calculate both windows. The short reserve protects the reject or defer loop. The long reserve tests whether scaling can prevent sustained shedding.

Use end-to-end measurements rather than nominal polling intervals. A ten-second metrics scrape can still produce a 35-second reaction if aggregation and control rollout are slow.

Keep the window tied to one action. Mixing a fast queue limit with a slow autoscaler produces a reserve that no controller can explain or test.

Measure net block growth

For every reaction window, count blocks allocated to active requests plus blocks admitted to reusable prefix storage.

Subtract blocks released by completed requests, expired entries, demotion, and eviction. The result is net block growth during the window.

Use physical blocks for the pool you are protecting. Token counts need conversion, and requests can have different cache layouts, dtypes, or hybrid states.

Keep active-request growth separate from retained-prefix growth. Active blocks cannot be discarded without changing scheduling behavior, while inactive prefix blocks are eviction candidates.

Record the positive tail of net growth across representative traffic. The p50 explains ordinary movement. A chosen p99 or p99.9 becomes the burst term if the service objective supports that percentile.

Do not sum per-second p99 values across the window. Calculate each complete window, then take a percentile of the resulting window totals. Summing marginal percentiles invents a burst that may never occur.

The top-ranking Spheron capacity guide covers block sizing and several ways to reduce bytes. Apply those changes before comparing historical windows across configurations.

Work a block-reserve example

Assume one replica exposes 120,000 usable KV blocks after weights, graphs, activations, and fixed runtime allocations are removed.

Its selected steady working set is 32,000 reusable blocks. The normal active-request pool reaches 62,000 blocks at the planning percentile.

The baseline is therefore 94,000 blocks. It leaves 26,000 blocks before the usable pool is full.

Now measure 20-second windows because the fast admission controller takes at most 20 seconds to detect pressure and apply a new limit.

The p99 positive net growth is 18,000 blocks. Add a measured ten percent telemetry and actuation allowance, or 1,800 blocks, for a reserve of 19,800.

Baseline plus reserve is 113,800 blocks. The replica retains 6,200 blocks of final slack inside the 120,000-block pool.

If each block occupies 256 KiB in this deployment, the 19,800-block reserve is about 4.83 GiB. That byte figure is an example, not a model constant.

Fanout's KV cache memory formula supplies the model-specific bytes per token. Multiply by tokens per physical block and verify the result against the engine's allocation.

Using the median 6,000-block burst would leave the system 12,000 blocks short of the measured p99. Average traffic is not a headroom policy.

Keep memory pools separate

Device headroom, KV free blocks, and the prefill activation reserve are not interchangeable labels.

The Elastic KV Cache paper shows that vLLM profiles a large prefill, reserves activation memory, and turns the remaining budget into a fixed KV block pool.

In its Qwen2.5-7B experiment, raising device utilization captured stranded device headroom but did not reclaim the separate activation reserve.

The same study found that a smaller prefill chunk recovered more KV capacity at almost the same median TTFT for its workload. Copy the method, not the setting.

Measure whether chunk size, tensor parallelism, CUDA graph capture, or another runtime change moves the usable block count before editing the burst reserve.

The current vLLM cache configuration also allows an explicit KV cache byte budget. A precise byte flag still needs a workload-derived reserve.

Changing gpu_memory_utilization until startup barely succeeds is not burst planning. It tests initialization, while the failure of interest happens under live arrivals.

Protect the reserve with control thresholds

Set a low-water threshold before the last reserved block. The controller needs room to observe the breach and finish its action.

For the example, admission control could tighten when free blocks fall below 19,800 and reject more aggressively below a smaller emergency floor.

Use hysteresis so the limit does not flip on every allocation and release. The recovery threshold should include enough blocks to survive another normal control interval.

Tie each response to the source of growth. Long prefills may need a chunk or concurrency limit. Long decodes may need output limits or fewer active sequences.

One-hit prefixes may call for faster demotion. A routing imbalance may call for moving new sessions instead of evicting useful local state.

The Aerospike tiering article explains the role of lower memory tiers. Offload helps only when transfer finishes inside the request's latency budget.

When demand exceeds service capacity for longer than the reaction window, headroom only delays failure. Admission control must reduce accepted work.

Test bursts instead of trusting the percentile

Replay the windows that created the p99 term. Preserve prompt lengths, output lengths, arrival spacing, cache identity, and session routing.

Measure free blocks, preemptions, rejected requests, evictions, cached tokens, TTFT, and inter-token latency throughout the replay.

Add a cold-replica case. A new replica may have active capacity but no prefix working set, so it trades cache pressure for extra prefill work.

Add a skewed-routing case. Fleet-wide free blocks can look healthy while one cache-affine replica crosses its reserve.

Fanout's goodput guide is the right continuation for judging whether the protected traffic still meets its latency targets.

Repeat the test after model, cache dtype, block size, prefix policy, routing, or batch limits change. Every one of those can move either the baseline or the burst tail.

Use a short worksheet

Record the usable KV blocks per replica after fixed memory pools are removed.

Choose the steady active pool and reusable working set from trace replay and service targets.

Name the control action, then measure its end-to-end reaction window.

Build rolling net-growth totals for that exact window and choose a percentile tied to the service objective.

Add measured event loss and actuation error, not a default safety percentage.

Verify that baseline plus reserve plus final slack fits the physical pool. If it does not, change capacity, policy, or admission before the burst arrives.

KV cache headroom for burst arrivals is a time budget expressed in blocks. It is defensible when the window, percentile, block bytes, and control action all come from the same deployment.