--- title: "Per-tensor vs per-head FP8 KV cache scales" description: "Compare per-tensor, static per-head, and dynamic per-token-head FP8 KV scales by range, metadata, calibration, and backend support." canonical_url: "https://fanout.sh/blog/per-tensor-vs-per-head-fp8-kv-cache-scales" md_url: "https://fanout.sh/blog/per-tensor-vs-per-head-fp8-kv-cache-scales.md" last_updated: "2026-09-05" access: "public" --- # Per-tensor vs per-head FP8 KV cache scales Compare per-tensor, static per-head, and dynamic per-token-head FP8 KV scales by range, metadata, calibration, and backend support. - Author: Suraj Gaud - Published: 2026-09-05 - Track: Inference engineering - Access: Public - Tags: FP8, KV cache, per-tensor scales, per-head scales, quantization, vLLM, FlashAttention, Triton Per-tensor vs per-head FP8 KV cache scales is not one binary choice. Current serving stacks use at least three scale layouts, and two of them are called per-head even though their storage costs differ sharply. A static per-head scale is one calibrated value per layer and KV head. A dynamic per-token-head scale adds new values for every cached token. Both isolate head ranges, but only the second grows with context length. Choose by naming the exact layout first. Then compare the range each scale covers, the bytes stored with the cache, the calibration path, and the attention backend that can consume it. ## Per-tensor vs per-head FP8 KV cache scales The current[vLLM quantized KV cache guide](https://docs.vllm.ai/en/stable/features/quantization/quantized_kvcache/)defines per-tensor scales as one Q, K, and V scale per attention layer. For K and V, every token and every KV head in that layer shares the same two values. One unusually large head can expand the scale for all the smaller heads. The same guide defines static per-attention-head scales. K and V then receive one scale per KV head, while Q receives one per query head. Those scales still stay fixed across requests and token positions. Per-head here means the calibration tensor has a head dimension. It does not mean the server writes a fresh scale next to every token. Current vLLM source also exposes fp8_per_token_head as a separate cache dtype. Its[quantization mode mapping](https://github.com/vllm-project/vllm/blob/32601ef7a1ce8aaa6d777778435ec499248906fb/vllm/v1/kv_cache_interface.py#L46-L103)classifies that mode apart from ordinary FP8. The per-token-head path measures each new K or V vector over its head dimension. Its range changes with both token and head, so it needs no fixed K or V scale from the checkpoint. That third layout is the expensive one in metadata. Calling both finer layouts simply per-head hides the operational difference. ## Range sharing explains the accuracy trade FP8 E4M3 has a wide floating-point range, but quantization still maps source values through a scale. Values outside the chosen range clip. Values far inside it use less of the available precision. Suppose one KV head reaches an absolute value of 32 while another stays below 2. A shared scale must preserve 32, so the smaller head occupies only one sixteenth of that represented range. A static per-head scale lets the second head use a range fitted to 2. It removes cross-head competition, but every token in that head still shares one value learned during calibration. A dynamic per-token-head scale goes further. A quiet token with an absolute maximum of 0.5 does not inherit the range of another token in the same head that reached 2. Finer range control does not guarantee a better model result. Quantization error passes through attention scores, softmax, and value aggregation, so the only useful verdict comes from the served workload. The[vLLM FP8 KV cache study](https://vllm.ai/blog/2026-04-22-fp8-kvcache)used uncalibrated per-tensor scale 1.0 for its main evaluations. Many tested paths stayed close to BF16 without finer scales. That result sets a sensible order. Test the cheap layout first. Pay for more scale resolution only when errors are systematic across prompts or context buckets. Fanout's[FP8 KV cache calibration guide](/blog/fp8-kv-cache-calibration)gives that control-and-treatment sequence in detail. ## Static per-head metadata is almost free Consider a model with 80 layers, 8 KV heads, and head dimension 128. Count only K and V because those are the tensors stored in the cache. Per-tensor calibration stores two FP32 scale values per layer. That is 80 times 2 times 4 bytes, or 640 bytes for the model. Static per-head calibration stores one K and one V scale for each head. The total is 80 times 8 times 2 times 4 bytes, or 5,120 bytes. The difference is 4,480 bytes. It is too small to affect cache capacity, even before comparing it with model weights and runtime buffers. At a 128K-token context, the FP8 K and V data for this layout occupies 2 times 80 times 8 times 128 times 131,072 bytes. That is exactly 20 GiB. Static per-head scale metadata does not change that 20 GiB calculation. Its cost is checkpoint complexity, calibration, and a narrower set of compatible kernels, not meaningful cache storage. The[KV cache memory formula](/blog/kv-cache-memory-formula-llm-inference)shows how to redo this calculation for another layer count, head geometry, precision, or tensor-parallel layout. ## Dynamic per-token-head scales add 3.125 percent Current vLLM's Triton backend packs one FP32 K scale and one FP32 V scale beside each token-head pair. The[backend layout code](https://github.com/vllm-project/vllm/blob/32601ef7a1ce8aaa6d777778435ec499248906fb/vllm/v1/attention/backends/triton_attn.py#L274-L304)adds eight scale bytes per pair. With head dimension 128, FP8 K and V data uses 256 bytes per token-head pair. The two scales add 8 bytes, so metadata adds 8 divided by 256, or 3.125 percent. For the same 80-layer, 8-KV-head, 128K-token example, the scales consume 80 times 8 times 131,072 times 8 bytes. That is 640 MiB. The cache is therefore about 20.625 GiB rather than 20 GiB. It remains close to half of the 40 GiB BF16 cache, but the scale overhead is no longer a rounding error. Head dimension changes the ratio. At dimension 64, data uses 128 bytes per token-head pair and the same scales add 6.25 percent. At dimension 256, they add 1.5625 percent. Dynamic scaling also performs a reduction over each new head vector, writes the scales, and reads them during attention. Memory arithmetic alone cannot predict whether that work pays back in latency. Benchmark cache capacity and inter-token latency together. Fanout's guide to[when KV cache quantization slows inference](/blog/when-kv-cache-quantization-slows-inference)explains why saved bytes can still lose on a short or unsupported path. ## Calibration changes which scale layout exists Uncalibrated vLLM FP8 uses scale 1.0. Its on-the-fly calibration option estimates static scales during warmup, while LLM Compressor can derive scales from a chosen dataset. Static per-head scales require representative calibration data because each head keeps its value after calibration. A short chat sample may miss the ranges reached by a long document or a different prompt template. Dynamic per-token-head scaling replaces that fixed K and V calibration with an online maximum for each vector. It adapts to the active request, but it also adds per-token work and cache-resident metadata. These approaches answer different failure modes. Static per-head helps when heads have consistently different ranges. Dynamic per-token-head also helps when range changes materially across token positions. Do not infer either pattern from one aggregate benchmark. Compare error or task score by head-sensitive model, prompt family, and context length, then reproduce the result on the exact serving path. If calibrated per-tensor already meets the quality threshold, a finer layout has no quality job left to do. Keep the simpler format unless a measured latency or compatibility result says otherwise. ## Backend support is the hard boundary The stable vLLM guide says static per-attention-head scaling requires LLM Compressor and the FlashAttention backend. A checkpoint with per-head arrays is not a portable promise across every vLLM backend. The current Triton backend lists fp8_per_token_head among its supported cache dtypes and stores its scale arrays inline. That is a different implementation path from loading static per-head checkpoint values. [TensorRT-LLM's attention documentation](https://nvidia.github.io/TensorRT-LLM/features/attention.html)currently describes a single-element KV scaling tensor and per-tensor quantization only. The same words therefore select different capabilities across engines. A deployment matrix should name engine version, attention backend, GPU, cache dtype, and scale source. Watch the startup logs and inspect the chosen kernel. A flag that is accepted but triggers a fallback is not the configuration that was benchmarked. Also keep weight precision separate. FP8 weights, FP8 activations, and an FP8 KV cache have different scale tensors and kernel paths. One can be per-head while another remains per-tensor. ## Use a three-stage decision Start with ordinary FP8 and scale 1.0 on a backend that supports it. Compare against BF16 using the same requests, seeds, sampling settings, and context buckets. If the quality gap is stable, calibrate per-tensor scales on production-shaped data. This keeps metadata fixed and retains the broadest backend compatibility. Move to static per-head scales when cross-head range differences remain, calibration data is trustworthy, and FlashAttention is the intended production backend. Try dynamic per-token-head FP8 when token-level range drift matters and the Triton path is acceptable. Charge the 3.125 percent scale overhead at head dimension 128 before calculating concurrency. For every candidate, record task quality, cache bytes, time to first token, inter-token latency, throughput, and the selected attention kernel. Per-tensor vs per-head FP8 KV cache scales is finally a choice among specific layouts. Static per-head costs kilobytes. Dynamic per-token-head costs a percentage of the cache. Backend support decides whether either can run. --- This representation contains public Fanout content only. Protected Pro lessons, account data, billing, checkout, and pricing are not included. Browse the public content map: https://fanout.sh/sitemap.md