FP8 KV cache layer skipping vs per-head scales

FP8 KV cache layer skipping vs per-head scales is a choice between two different fixes. Per-head scales keep the cache in FP8 and change its calibration granularity. Layer skipping keeps selected caches in BF16.

Their memory costs differ sharply. Static per-head scales add a small fixed tensor per layer. Skipping a layer adds one byte for every retained key and value element in that layer.

That does not make per-head scales the automatic answer. A sensitive attention type can still need BF16, while a model with head-specific ranges may recover with finer scales and retain almost all of FP8's capacity.

FP8 KV cache layer skipping vs per-head scales

Current vLLM KV cache documentation defines per-tensor FP8 with one scale each for Q, K, and V.

Its per-attention-head path assigns one Q scale per query head and one K and V scale per KV head. This path currently requires Flash Attention and calibration through llm-compressor.

The layer-skip flag operates on another axis. It leaves chosen layer indices or attention types at the model's native dtype while all other eligible layers use FP8.

Per-head scaling changes how values map into E4M3. It does not change the one-byte FP8 payload stored for each K or V element.

Skipping a BF16 layer does change that payload. Every cached K and V element in the skipped layer uses two bytes instead of one.

The two controls can be combined. Calibrate per head for the quantized layers, then leave only the layer group that still fails its quality or latency target in BF16.

Per-head scales have fixed overhead

Suppose a layer has H query heads and Hkv KV heads. The documented per-head shapes contain H plus two times Hkv scales, compared with three scales under per-tensor calibration.

If each scale is float32, the extra static bytes are 4 times (H + 2Hkv - 3). They do not multiply by sequence length, batch size, or KV block count.

For 64 query heads and 8 KV heads, the documented shapes contain 80 scale values. That is 320 bytes per layer, or 308 bytes more than three float32 scalar scales.

Across 24 layers, the difference is 7,392 bytes. Tensor-parallel loading can shard or reduce the resident arrays, so treat this as a simple upper-bound worksheet, not an allocator measurement.

The current vLLM compressed-tensors implementation stores static scales as float32 parameters.

This static per-head scheme is different from per-token-head quantization. A dynamic scale stored for every token and KV head would grow with the cache. Do not use that layout's metadata formula here.

For the static scheme, scale storage is rarely the capacity decision. Calibration time, backend support, and measured accuracy matter more than a few kilobytes.

Layer skipping follows retained-token footprint

Count retained token-layer positions before counting skipped layers. A full-attention layer at context T retains T positions. A sliding-window layer with window W retains at most W.

Let A be the sum of retained positions across every cache layer. Let S be the same sum over only the skipped layers.

Ignoring block padding and allocator overhead, all-FP8 cache bytes are C times A. C includes K and V, KV heads, and head dimension at one byte per element.

Mixed FP8 and BF16 bytes are C times (A + S). The skipped positions add one more byte per element because BF16 uses two bytes where FP8 uses one.

The compression ratio against an all-BF16 cache is 2A divided by (A + S). The fraction of skipped layers is not enough to compute it.

If half the layers are skipped and every layer retains the same number of tokens, S equals half of A. Compression falls from 2x to 1.33x.

If the skipped half contains 128-token windows while the other half retains 32,768 tokens, S is a tiny fraction of A. Compression stays close to 2x.

Fanout's KV cache memory formula gives the full K and V sizing terms that are bundled into C here.

Work the gpt-oss-20b numbers

The official gpt-oss-20b configuration has 24 layers that alternate sliding and full attention.

It also has 8 KV heads, head dimension 64, and a sliding window of 128 tokens. That gives 12 full-attention layers and 12 sliding-window layers.

Use a 32,768-token context and one sequence. The full layers retain 12 times 32,768, or 393,216 token-layer positions.

The local layers retain 12 times 128, or 1,536 positions after their windows fill. Total A is 394,752, while S for skipping every local layer is 1,536.

Each token-layer position stores K and V across 8 heads of width 64. BF16 therefore uses 2 times 8 times 64 times 2 bytes, or 2,048 bytes per position.

The all-BF16 cache is about 771 MiB. All FP8 is about 385.5 MiB. FP8 with the 12 sliding layers kept in BF16 is about 387 MiB.

Skipping half the layers gives back only 1.5 MiB in this example. The resulting compression ratio is about 1.992x, not 1.33x.

Layer count is a poor proxy for memory here. Leaving small local caches in BF16 can preserve almost the full capacity gain.

Block alignment and page padding can move the measured allocation. Use the formula to predict direction, then confirm the engine's reported cache capacity.

Performance can favor the larger cache

Layer skipping can improve speed even though it stores more bytes. Quantization has kernel work, and a 128-token local window may be too small to amortize it.

The vLLM FP8 study measured gpt-oss-20b on H100 with BF16, all-FP8, and FP8 that skipped sliding-window layers.

Its fitted inter-token-latency slope was 8.94e-06 ms per token for BF16, 7.14e-06 for all-FP8, and 6.34e-06 for the skip variant.

The same study reported a break-even point near 22,109 tokens for all-FP8 and 7,659 tokens when the 128-token sliding layers stayed in BF16.

For a Gemma model with a 512-token window and head dimension 256, the study found enough local work to make quantizing the sliding layers useful. The right skip set is model and backend specific.

The memory arithmetic shows the capacity cost. A latency sweep shows whether that cost removes more quantization overhead than it adds memory traffic.

Use the failure pattern to choose the knob

Try calibrated per-head scales when the FP8 error comes from different numerical ranges across heads, while the same layer types remain healthy.

The calibration set must represent the deployment workload. Finer granularity cannot repair a scale learned from inputs that omit the long contexts or task distribution that fail in production.

Fanout's FP8 KV cache calibration guide covers held-out quality checks and context-length sweeps.

Try layer skipping when failures cluster by layer index or attention type. A hybrid model may have local layers whose accuracy or kernel cost behaves differently from its full-attention layers.

If one layer group fails and per-head calibration fixes it, keep every layer in FP8. If the group still fails, keep that group in BF16 and preserve FP8 elsewhere.

If failures appear only at long contexts, do not test one short perplexity run. Include retrieval, generation, and task metrics across the intended context range.

Run a four-way ablation

Start with an all-BF16 baseline. Record quality, time to first token, inter-token latency, throughput, and cache capacity.

Then test uncalibrated per-tensor FP8. This separates the raw format and kernel path from the calibration improvement.

Next test calibrated per-head FP8 with the same model, backend, traffic trace, and memory limit. Record both quality recovery and any calibration-specific runtime change.

Finally test per-head FP8 with the suspected layer group skipped. This shows whether BF16 is still buying quality or speed after the scale problem is fixed.

• Keep batch shape and context distribution fixed across all four runs.
• Report cache capacity in tokens as well as allocated bytes.
• Break results out by context bucket and task, not only one average.
• Inspect each attention type separately when the model is hybrid.

Do not compare a calibrated skip run against an uncalibrated all-FP8 run and credit the difference to skipping. Change one axis at a time.

The decision rule

Try per-head scales first when the backend supports them. Their static metadata does not grow with the cache, and they preserve the FP8 payload in every layer.

Layer skipping is the targeted fallback when a layer group still fails quality or wastes time in its quantized kernel. Price it with retained token-layer positions, not the percentage of layers skipped.

For small sliding windows, the BF16 penalty can be almost invisible beside full-attention cache memory. The gpt-oss example keeps about 1.992x compression while leaving half its layers unquantized.

Use both controls when needed: per-head calibration for the FP8 layers, and BF16 only for the layers that the ablation proves should stay there.