FP8 attention vs FP8 KV cache

FP8 attention vs FP8 KV cache is a comparison between arithmetic and storage. An FP8 KV cache stores old keys and values in one byte each. FP8 attention also runs the matrix multiplications that consume them in FP8.

Those settings can travel together, but they are not interchangeable. A backend may store FP8 cache pages and convert them before attention. That halves cache capacity without guaranteeing faster attention.

The check is simple: calculate the saved bytes, identify the actual attention kernel, then measure the context length where lower memory traffic repays quantization overhead.

FP8 attention vs FP8 KV cache

During autoregressive generation, each layer appends one key and one value per new token. The KV cache keeps those tensors so later tokens do not recompute the full prefix.

An FP8 KV cache changes the dtype of those stored tensors. Moving from BF16 at two bytes per element to FP8 at one byte cuts the raw K and V storage by 50 percent.

FP8 attention changes the operations that read the cache. The two large products are query times key transpose and softmax probabilities times value.

If those products run on FP8 inputs with supported tensor core instructions, the kernel can reduce both memory traffic and arithmetic cost. It still needs higher precision for operations such as softmax and accumulation.

The FlashAttention-3 paper uses block quantization and incoherent processing to control that error. Its FP8 path reached nearly 1.2 PFLOPS on H100 in the paper's attention benchmarks.

Model weight precision is a third decision. An FP8 checkpoint can keep a BF16 cache, while a BF16 checkpoint can use an FP8 cache.

Fanout's quantization comparison separates weight formats from cache formats.

Calculate the cache saving first

KV bytes per token equal 2 times layers times KV heads times head dimension times bytes per element. The first factor stores one key and one value.

The Llama 3 report lists 80 layers, 8 KV heads, and a head dimension of 128 for Llama 3 70B.

With BF16, the calculation is 2 times 80 times 8 times 128 times 2 bytes. That is 327,680 bytes, or 320 KiB, per token.

With FP8, the last factor falls from 2 bytes to 1. The result is 163,840 bytes, or 160 KiB, per token.

A 32,768-token sequence therefore needs 10 GiB of BF16 KV data or 5 GiB of FP8 KV data. This excludes allocator metadata, temporary buffers, and cache blocks shared across requests.

The saving is real even if the attention math remains BF16. Five freed GiB can hold another long request, a larger batch, or more prefix-cache entries.

Fanout's KV cache formula guide derives the same calculation for other layer, head, and dtype choices.

Storage precision does not select the kernel

The serving engine chooses an attention backend from the GPU, model architecture, cache dtype, head dimension, and enabled features. A command-line dtype alone does not describe the executed path.

In a 2024 vLLM support issue, enabling an FP8 cache could force a different backend because the FlashAttention path then in use did not support that cache format.

That older path still saved memory, but quantization and dequantization overhead could erase latency gains. It is evidence for checking the installed engine rather than treating an FP8 cache as an FP8 kernel.

The current validated Hopper path is different. The vLLM FP8 study uses its FlashAttention-3 fork with native FP8 cache support and FP8 attention.

The same study uses FlashInfer on B200. NVIDIA's TensorRT Edge-LLM documentation describes another Blackwell kernel that consumes FP8 Q, K, and V directly.

Record the engine version, GPU, kernel, model shape, and cache dtype. "FP8 enabled" omits most of the performance contract.

Find the latency break-even point

Quantizing the new K and V values, calculating scales, and setting up the low-precision path have fixed costs. Short contexts may finish before lower cache bandwidth repays them.

The vLLM study fit inter-token latency against context length. For Llama 3.1 8B on H100, its improved FP8 path reached break-even near 7,010 tokens.

Past that point, the FP8 inter-token-latency slope was 54 percent of the BF16 slope. At concurrency 8, the study measured 14.9 percent more output throughput.

Those numbers belong to that H100, model, engine version, and benchmark. They do not predict a different GPU or backend.

Run a context sweep instead. Hold prompt distribution, output length, concurrency, scheduler settings, and model weights constant. Compare BF16 cache with FP8 cache on the backend reported in the engine logs.

Record time to first token, inter-token latency, output throughput, cache capacity, and accuracy. A capacity win can be useful even when single-request latency is flat.

Fanout's inference engineering guide covers the workload controls needed for a serving comparison.

Treat accuracy as a backend property

One FP8 scale for a whole tensor can lose small values when outliers set the range. Per-head or finer-grained scales can preserve more signal but add metadata and calibration work.

Accumulation is another source of error. The vLLM team found that an early Hopper FP8 FlashAttention-3 path fell from 91 percent to 13 percent on a 128k needle test before its accumulation fix.

That result is not a verdict on every FP8 cache. It shows that storage dtype, attention kernel, scale choice, and accumulation method form one accuracy path.

After fixes, vLLM reported 97 to 98 percent recovery of baseline AUC for Llama 3.3 70B on its 128k MRCR evaluation. Other models and backends showed larger gaps.

Calibration deserves a workload test. The same study found a consistent loss for uncalibrated Kimi K2.5 with FlashMLA and recommends calibration when the target evaluation drops below the acceptable floor.

Test long-context retrieval and the product's real tasks. Perplexity alone can miss a failure that appears only after tens of thousands of cached tokens.

Watch head dimension and hybrid attention

FP8 does not help every layer equally. Short sliding-window layers read much less cache than full-attention layers, so fixed quantization overhead can dominate.

In vLLM's gpt-oss-20b results, skipping FP8 for sliding-window layers moved the decode break-even from about 22,109 tokens to 7,659 tokens.

Head dimension also changes kernel behavior. The study warns that head dimension 256 can make long-context prefill about 1.6 times slower when the accuracy-preserving accumulation path is enabled.

That creates a mixed policy: keep small-window or awkward-shape layers in BF16 while quantizing the full-attention cache that drives capacity and bandwidth.

Do not infer this split from the model name. Inspect the attention configuration, confirm backend support for each layer type, and verify that the engine did not fall back to a slower kernel.

Choose from measured outcomes

Use an FP8 KV cache when cache capacity is the constraint and target-task accuracy survives. The 50 percent raw memory reduction applies even if the kernel converts values before arithmetic.

Require an FP8 attention path when the goal is lower decode latency. Confirm it in backend logs, then locate the context-length break-even with a sweep.

Keep BF16 for layers or models whose quantization overhead, head shape, or accuracy loss does not repay the saved bytes. A hybrid cache policy can beat one global dtype.

The decision is not FP8 attention or FP8 KV cache. It is whether the chosen backend turns compressed storage into supported low-precision math without giving back the gain in conversion cost or accuracy.