KV cache scale calibration under drift

KV cache scale calibration under drift fails quietly. The model still serves requests and memory use still falls.

A benchmark from the calibration domain can also pass while live keys and values exceed the ranges used to choose static scales.

The fix is not to make every scale larger. Wider coverage reduces saturation but spends precision on rare values. Operators need a replay that shows which layers and heads leave the calibrated range, then ties that evidence to task quality.

Static scales are deployment artifacts. Treat them like a model version: record their data, context length, recipe, and evaluation envelope.

KV cache scale calibration under drift

The current vLLM quantized KV-cache guide supports FP8 scales per tensor or per attention head.

Without calibration, vLLM sets every scale to 1.0. Its recommended path estimates scales from a dataset with LLM Compressor.

The documented example takes 512 shuffled UltraChat samples, truncates them to 2,048 tokens, and writes the calibrated scales into the saved model.

Those choices define an envelope. A code workload, another language, a 64K retrieval prompt, or a new chat template can produce different key and value ranges.

This is calibration-data drift. It means serving activations no longer resemble the data that fixed the quantization parameters.

It is different from ordinary traffic-volume drift. Ten times more requests do not change one token's numeric range, although they make a rare failure happen more often.

It is also different from model-weight drift. A new checkpoint changes the projections themselves and should invalidate the old calibration unless equivalence is proven.

Fanout's per-tensor versus per-head guide explains the scale shapes. The operational question is whether those saved values still cover live activations.

A static scale freezes observed range

Calibration runs representative tokens through the full-precision model and observes activation ranges. The quantizer uses those observations to map values into FP8.

For one layer and head, call the largest calibrated magnitude A. A static scale chosen from A can represent values near that range while retaining useful resolution for smaller values.

If production later produces a magnitude above A, the quantizer can saturate at the largest value covered by the saved scale. The excess is clipped rather than stored exactly.

Suppose A is 8 and a live token produces 10 in that same group. If the effective covered range still ends at 8, the reconstructed value is at most 8.

The absolute error is 2. Relative to the live value, that is 20% error before the attention dot product mixes it with other dimensions.

Increasing the scale to cover 12 avoids this example's saturation. It also assigns more of FP8's finite representational range to large magnitudes, changing rounding error for the ordinary values below 8.

FP8 spacing is not uniform like a simple integer grid, so one linear step-size formula cannot describe every value. The tradeoff remains: range and precision share a finite format.

Measure saturation or overflow by group. A fleet-wide maximum hides whether one attention head is repeatedly outside range while the rest remain well covered.

Per-head scales localize the mismatch

One per-tensor K scale must cover every KV head in that layer. A single outlier-heavy head can therefore set the range for quieter heads.

Per-head calibration gives each KV head its own K and V scale. That keeps one wide distribution from spending the precision budget of every neighboring head.

The official LLM Compressor per-head example calls 512 samples a starting point.

It also notes that more samples can improve accuracy.

More samples only help if they cover the right workloads. Ten thousand short English chats still do not test a long multilingual retrieval trace.

Per-head scales also do not eliminate drift. They make its location visible and reduce cross-head coupling, but a live head can still exceed its own calibrated range.

Compare the live maximum to the calibrated maximum for K and V separately. Keys and values need not have the same outlier pattern or quality sensitivity.

Fanout's layer-skipping comparison covers another boundary. Keeping a sensitive layer in BF16 removes scale error there instead of widening every FP8 group.

Use that escape hatch when a small set of layers fails repeatedly across representative calibration datasets.

Context length can create hidden drift

The vLLM example truncates calibration samples at 2,048 tokens. That is a reproducible starting point, not proof for every configured context length.

A 32K production request contains positions, document structures, and attention histories that never appear in a 2K calibration pass.

Even if most early-token ranges match, later positions can produce different head-level extremes. A short perplexity test will not exercise those cache entries.

Build evaluation slices by prompt length. Include short chat, the normal production median, the p95 length, and the longest length admitted by the server.

Do the same for output length. Reasoning and code generation can append thousands of new KV entries after the prompt, so prompt-only coverage is insufficient.

The KVarN paper attributes low-bit reasoning degradation primarily to incorrect token scales and reports error accumulation across generation steps.

KVarN is a calibration-free method, not a direct test of vLLM's static FP8 path. Its result still warns against validating only the first few generated tokens.

Run exact-match or pass-rate evaluations over full outputs. A response that begins correctly can diverge after quantization errors alter one token and every later token conditions on it.

Static and online scales fail differently

Static calibrated scales add almost no per-token metadata and avoid calculating a new range for every cache entry. Their risk is a stale envelope.

Online quantization measures the current group and can adapt its scale to each token and head. Its risk is added work and repeated scale metadata.

LMDeploy's KV-cache quantization guide is one production example of online per-head, per-token INT4 and INT8 quantization.

Do not transfer its performance claims to vLLM FP8. The formats, kernels, hardware paths, and scale layouts differ.

The comparison is architectural. Static scales pay calibration and drift risk. Online scales pay runtime range calculation, metadata, and kernel complexity.

Fanout's KV-cache memory formula supplies the unquantized baseline. Add the exact scale bytes and allocator layout before claiming a capacity ratio.

A hybrid policy is also possible. Keep static per-head FP8 for covered layers and native precision for the layers that fail the drift replay.

Choose from measured quality and throughput together. A dynamic scheme that preserves accuracy but erases decode speed may not improve useful capacity.

Do not confuse two kinds of distribution drift

Calibration drift describes a mismatch between the data used to choose quantization parameters and the activations produced by serving traffic.

Quantization can also distort the model's output distribution even when the input workload matches calibration.

The DGAP paper studies structured local misranking in low-bit KV-cache quantization. It calls the resulting change in top-K logits local distribution drift.

That downstream drift is an effect of quantization error. It is not evidence that the calibration dataset changed.

Keep the terms separate in incident reports. One asks whether live activations exceeded a saved envelope. The other asks whether compressed attention changed token ranking.

They can occur together. Out-of-range KV values can increase attention error, which can then move close logits across a ranking boundary.

Measure both sides when possible: activation saturation by layer and head, then task accuracy or logit divergence on the same trace.

Do not rely on generated prose inspection alone. Small ranking changes are hard to see by eye and can be severe on retrieval, code tests, or exact-match reasoning.

Build a production-shaped calibration set

Start from traffic classes rather than a generic corpus name. Sample the languages, prompt templates, tools, code, retrieval documents, and reasoning styles the endpoint actually serves.

Stratify by context length and output length. Rare long requests can dominate KV-cache risk even when they are a small share of request count.

Preserve chat templates and special tokens. Calibration inputs should pass through the same serialization used in production.

Hold out a drift set that is deliberately different. Include a new domain, unusually long contexts, symbol-heavy code, and the least common supported language.

For each layer and KV head, record calibrated maxima, live maxima, saturation counts, and the fraction of tokens outside the saved range.

Then run full-precision and FP8 versions on the same prompts. Compare retrieval accuracy, exact match, code tests, perplexity where it is meaningful, and output divergence by generation position.

Sweep per-tensor and per-head calibration. If a small number of layers dominate the failures, repeat with those layers skipped from KV quantization.

The result should name a supported envelope. “FP8 passed” is weaker than “FP8 per-head passed these workloads through 32K input and 4K output with these quality bounds.”

Recalibrate on a versioned trigger

Recalibrate when the model checkpoint, tokenizer, chat template, supported context length, or traffic mix changes materially.

Also recalibrate when live range telemetry crosses a reviewed threshold, even if aggregate quality has not yet moved. Saturation is an early warning, not a complete SLO.

Version the dataset manifest, seed, sample count, maximum sequence length, quantization recipe, library versions, and saved scale tensors.

Canary the new artifact against the old one. Hold model weights and serving load constant so a scale change is not confused with a scheduler or kernel change.

Compare quality, TTFT, TPOT, throughput, cache capacity, and preemption rate. Better numeric coverage is not a win if the deployed path loses the concurrency that motivated FP8.

Keep a rollback artifact. Static scales are small, but their effect reaches every cached token produced by the deployment.

KV cache scale calibration under drift is controlled when the saved range matches production-shaped activations, long outputs remain correct, and the artifact is regenerated whenever its envelope changes.