--- title: "Per-token-head KV cache scale overhead" description: "Calculate the FP32 scale bytes stored beside per-token-head FP8 and INT4 KV data, then compare actual capacity with static per-head calibration." canonical_url: "https://fanout.sh/blog/per-token-head-kv-cache-scale-overhead" md_url: "https://fanout.sh/blog/per-token-head-kv-cache-scale-overhead.md" last_updated: "2026-10-10" access: "public" --- # Per-token-head KV cache scale overhead Calculate the FP32 scale bytes stored beside per-token-head FP8 and INT4 KV data, then compare actual capacity with static per-head calibration. - Author: Suraj Gaud - Published: 2026-10-10 - Track: Inference engineering - Access: Public - Tags: KV cache, quantization, vLLM, inference memory Per-token-head KV cache scale overhead is usually left out of capacity estimates. FP8 is called half the memory of BF16. INT4 is called half the memory of FP8. Neither estimate includes the scale stored beside each quantized group. That omission is small but systematic. In a dynamic per-token-head layout with one FP32 scale for K and one for V, the scale cost is 8 bytes per KV head, per layer, per retained token. For a 32-layer model with 8 KV heads and a head size of 128, FP8 KV data costs 64 KiB per token. Its scales add 2 KiB. The real total is 66 KiB, so the effective compression against BF16 is 1.94x rather than 2x. ## Per-token-head KV cache scale overhead Start with the ordinary KV cache formula. For one retained token, the unquantized payload is: 2 × layers × KV heads × head size × bytes per element. The factor of two represents keys and values. This is the same accounting developed in Fanout's[KV cache formula guide](/blog/kv-cache-formula-llm-inference-memory). Dynamic per-token-head quantization adds two scale values for every layer and KV head at every token position. If each scale is FP32, the metadata is: 2 × layers × KV heads × 4 bytes. The scale term does not contain head size. It grows with layers, KV heads, and retained tokens, but not with the number of elements inside one head. “Per-head” can describe two different layouts. A static calibrated scale may be stored once per head. A dynamic per-token-head scale is stored again at every token position. ## The FP8 scale tax is 4 divided by head size FP8 uses one byte per element. Its K and V payload per layer and KV head is 2 × head size bytes. The two FP32 scales add 8 bytes. Divide metadata by payload and the layer count, head count, and token count cancel: 8 ÷ (2 × head size) = 4 ÷ head size. At a head size of 64, scales add 6.25%. At 128, they add 3.125%. At 256, they add 1.5625%. The ratio describes layout, not benchmark performance. It counts bytes before allocator rounding, temporary buffers, fragmentation, or attention-kernel workspace enter the picture. The current[vLLM Triton attention backend](https://github.com/vllm-project/vllm/blob/main/vllm/v1/attention/backends/triton_attn.py)makes the layout concrete. For its dynamic per-token-head mode, each cache slot contains K data, one padded FP32 K scale, V data, and one padded FP32 V scale. The scale tensor is indexed by block, token position, and KV head. ## Work a 32-layer GQA model Consider a model with 32 layers, 8 KV heads, and a head size of 128. For one retained token, FP8 K and V payload is: 2 × 32 × 8 × 128 × 1 byte = 65,536 bytes, or 64 KiB. The dynamic scale cost is: 2 × 32 × 8 × 4 bytes = 2,048 bytes, or 2 KiB. The actual per-token KV allocation is therefore 66 KiB before block-level rounding. A BF16 cache for the same model is 128 KiB per token. The effective compression ratio is 128 ÷ 66, which is about 1.94x. At a 128 Ki-token context, the FP8 data occupies 8 GiB. The dynamic scales occupy another 256 MiB, bringing the total to 8.25 GiB. BF16 would occupy 16 GiB. A capacity plan that assumes exactly 8 GiB for this FP8 cache is already short by 256 MiB. That gap can be enough to change the number of sequences admitted to a tightly packed worker. ## Static per-head calibration pays once vLLM also documents static per-head scales. Its[quantized KV cache guide](https://docs.vllm.ai/en/v0.31.0/features/quantization/quantized_kvcache/)distinguishes per-tensor scales from per-attention-head scales. In a static per-head layout, one K scale and one V scale can be calibrated for each KV head in each layer. For the example model, that is still 2,048 bytes, but it is 2,048 bytes for the model rather than for every token. At 128 Ki tokens, the dynamic layout repeats those 2 KiB of scales 131,072 times. The static layout does not. Its metadata is negligible compared with the 8 GiB FP8 payload. Dynamic and static scales also differ numerically. Dynamic scales track each token and head. Static scales depend on calibration data and must cover all values expected at serving time. Fanout's comparison of[FP8 layer skipping and per-head scales](/blog/fp8-kv-cache-layer-skipping-vs-per-head-scales)explains why a smaller scale group and selective high-precision layers solve different error modes. ## INT4 makes the metadata fraction larger Four-bit data halves the payload again, but two FP32 scales still consume 8 bytes. For packed INT4 K and V, the payload per layer and KV head is one head-size worth of bytes. The metadata-to-payload ratio becomes: 8 ÷ head size. At a head size of 128, two FP32 scales add 6.25% to packed INT4 data. At 64, they add 12.5%. The same vLLM backend handles its INT4 per-token-head layout by halving the packed K and V widths, then adding space for the two FP32 scale values. INT4 keeps its memory advantage. But “half of FP8” is an incomplete capacity estimate when FP8 and INT4 carry the same scale precision and grouping. Quantization schemes can also store zero points or other group metadata. Price the exact kernel layout rather than transferring the two-scale result to a different format by name. ## Head size changes percentage, head count changes bytes The percentage overhead depends on head size because the scale count stays fixed while the payload inside each head changes. The absolute overhead depends on KV-head count. Doubling KV heads doubles both payload and scale bytes, so the percentage stays constant but the bytes per token double. When comparing multi-head attention with grouped-query attention, GQA reduces both quantized data and per-head metadata in the same proportion. Layer skipping changes the accounting differently. If selected layers keep BF16 KV data, those layers have no quantization scales but use two bytes per element. Calculate each layer group separately, then add the totals. That mixed-precision arithmetic belongs beside accuracy measurements. A single global “FP8 cache” label hides both the skipped layers and the scale granularity. ## Convert bytes into block capacity Serving engines reserve KV cache in blocks rather than as one perfectly packed token array. After calculating bytes per token, multiply by the block size and round to the allocation unit used by the backend. For the example model, a 16-token block contains 1,056 KiB of FP8 data plus scales. That is 66 KiB × 16, before any additional alignment imposed by the allocator. Then divide the usable cache pool by the rounded block bytes. Do not divide total GPU memory directly. Weights, activations, CUDA graphs, communication buffers, and runtime headroom all compete for the same device. A measured block count is the final capacity proof. The formula catches missing metadata. The initialized engine reports what survived every rounding and reservation. ## Choose granularity from accuracy and capacity together Per-token-head scales can recover range where one static scale is too coarse. Their cost is predictable: 8 bytes per layer, KV head, and token when K and V each use one FP32 scale. For FP8, divide 4 by head size to get the metadata fraction. For packed INT4 with the same two scales, divide 8 by head size. Use those ratios as the first correction to the dtype-only estimate. Then add mixed-precision layers, zero points if present, block rounding, and runtime reserves from the implementation you will actually deploy. The corrected block count matters when it changes concurrency, maximum context, or the margin that prevents preemption under load. --- This representation contains public Fanout content only. Protected Pro lessons, account data, billing, checkout, and pricing are not included. Browse the public content map: https://fanout.sh/sitemap.md