FP8 KV cache calibration: a practical test

FP8 KV cache calibration should begin with a control, not a calibration dataset. Run the model with scale 1.0, compare it with BF16 on the real workload, and calibrate only if the loss is consistent.

That FP8 KV cache calibration order matters because uncalibrated FP8 now works well on many supported vLLM paths. A generic calibration set can add work without proving that it fixes the model, backend, and context lengths you serve.

Establish the BF16 result first. Then bucket FP8 quality by sequence length and estimate scales from representative data only if the gap stays negative.

What FP8 KV cache calibration changes

An FP8 cache stores each key and value with fewer bits than BF16. A scale maps the tensor's real range into the values that FP8 E4M3 can represent.

The TensorRT Edge-LLM documentation gives the basic calculation. It collects the largest absolute value, called amax, for K and V in each layer.

The quantization scale is amax divided by 448, the largest finite E4M3 value used by that path. Inference stores FP8 values and supplies the saved K and V scales to the attention kernel.

If the scale is too small, values above the represented range clip. If it is too large, ordinary values use a narrower share of the available FP8 levels and rounding error grows.

Calibration observes actual activations so the scale reflects a workload rather than a convenient constant.

This only changes the cache and attention path. Weight calibration is a separate choice. Fanout's FP8 format comparison separates weight, activation, and cache formats.

Start with scale 1.0

Current vLLM quantized KV cache documentation offers an uncalibrated path where every scale is 1.0.

That sounds crude, but it is a useful baseline. It has no calibration corpus, no preprocessing choice, and no stale scale file inherited from another model revision.

The 2026 vLLM FP8 KV cache study deliberately tested per-tensor scale 1.0 as its accuracy control.

Across the validated Hopper and Blackwell paths, many models stayed close to BF16 even on reasoning and long-context evaluations. That makes uncalibrated FP8 a testable starting point, not an automatic production default.

Run the same model revision and prompts twice. Keep sampling deterministic where the engine permits it, and change only the KV cache dtype and attention path.

Record task score, output validity, time to first token, inter-token latency, and peak cache capacity. A quality comparison without the systems gain cannot tell you whether FP8 is worth keeping.

Fanout's KV cache quantization guide covers the memory saving and the accuracy mechanisms behind this control.

Bucket quality by context length

One aggregate score can hide the failure that matters. A small average change may combine clean short prompts with a steady decline after 32,000 tokens.

Split evaluation examples into context buckets that match the service: perhaps under 8K, 8K to 32K, 32K to 64K, and above 64K. Use the boundaries your traffic supports rather than copying these numbers.

Within each bucket, compare BF16 and FP8 on the product outcome. For retrieval, check whether the answer uses the right passage. For tool use, check the arguments. For code, run the tests.

Perplexity is useful as an early warning, but it averages token prediction error. It does not prove that a long answer preserves a fact, a tool call remains valid, or a reasoning chain reaches the same result.

The vLLM study found a useful calibration signal on Kimi-K2.5 with FlashMLA. The uncalibrated score shifted downward across sequence-length buckets instead of failing in a few noisy samples.

That pattern is stronger evidence than one bad prompt. Re-run enough examples to separate a persistent shift from sampling variance and benchmark noise.

Fanout's post on when KV cache quantization slows inference explains why backend and workload must stay in the test matrix.

Calibrate on the distribution you serve

If scale 1.0 shows a repeatable quality gap, build the calibration set from production-shaped inputs. Match language, prompt template, task family, and context-length distribution.

Do not calibrate a 64K document service on 2K chat snippets merely because a public chat dataset is easy to download. The collected extrema will describe the easy dataset, not the cache values produced by the service.

The official LLM Compressor example uses 512 UltraChat samples with a maximum sequence length of 2,048.

Treat those values as an executable example. They are not evidence that 512 short chat samples cover every model or long-context workload.

Sample across the same prompt-length buckets used in evaluation. Keep calibration and evaluation examples disjoint so the final comparison measures generalization rather than reuse.

Freeze the tokenizer, chat template, model revision, quantization recipe, and random seed. A scale file has meaning only with the exact model path that produced its K and V activations.

After calibration, rerun both the BF16 control and the scale-1.0 FP8 baseline. The calibrated result must beat the uncalibrated result where it was weak without creating a new regression elsewhere.

Choose per-tensor or per-head scales

Per-tensor calibration assigns one K scale and one V scale to a tensor. It is simple, compact, and supported by more attention paths.

Per-head calibration gives each attention head its own scale. A head with a narrow activation range no longer shares a scale set by another head's outlier.

The current vLLM documentation exposes both strategies, but per-attention-head scales require LLM Compressor and the FlashAttention backend.

That support boundary is part of the decision. A theoretically finer scale is useless if the production backend silently falls back, rejects it, or runs a different kernel from the benchmark.

Compare per-tensor and per-head only after per-tensor calibration leaves a measurable gap. The finer option adds scale metadata and another backend constraint, so it should earn its place in the matrix.

Also inspect layer types. vLLM lets deployments leave named layers, such as sliding-window attention layers, at native precision while quantizing the rest.

Selective BF16 can be a cleaner fix than forcing one calibration rule across attention mechanisms with different ranges and kernels.

Validate the checkpoint inside the serving path

The cache format does not run alone. Attention backend, GPU generation, head dimension, prefix caching, chunked prefill, and model architecture all affect the executed path.

An older vLLM scale issue shows why version context matters. FP8 cache support existed, but scale export and compatible append kernels were still incomplete in that release window.

Do not carry its operational advice into a current deployment. Do carry forward the lesson: verify the engine version and kernel selected by the actual server.

Run long prefill, long decode, and mixed concurrent traffic. The vLLM study reports that short contexts can lose to BF16 because a fixed FP8 overhead has not yet amortized.

It also identifies head dimension 256 as a path where long-context prefill can regress even when decode improves. A single tokens-per-second number would miss that split.

Check numerical quality after the server loads the checkpoint. The serving engine must discover and apply the intended scales.

A stop rule for FP8 KV cache calibration

Keep scale 1.0 when it matches BF16 within the product's quality tolerance and delivers the required memory or latency gain on the production backend.

Calibrate when the FP8 gap is repeatable across examples or sequence-length buckets, especially on a less-tested attention path. Use data shaped like the workload and rerun the same control.

Try per-head scales or selective BF16 layers only when per-tensor calibration leaves a localized problem and the deployed backend supports the alternative.

Reject FP8 for that path when calibrated quality still misses the threshold, the backend falls back, or the workload is too short or prefill-heavy to recover the added overhead.

A scale file is useful only when one exact model, engine, GPU, and workload keeps its quality while the smaller cache pays for itself.