KV cache quantization: memory savings by bits
KV cache quantization stores the keys and values from earlier tokens with fewer bits. Moving from BF16 to FP8 roughly halves the raw cache. Four-bit storage cuts the raw tensor to one quarter.
Those ratios are only the starting point. Scales, zero points, full-precision residual tokens, unsupported layers, and quantization kernels all change the memory and speed you get in practice.
The useful calculation is how many bytes remain for your model, context length, batch size, and runtime.
What KV cache quantization changes
Autoregressive decoding saves a key vector and a value vector for every cached token at every attention layer. The next token reads those vectors instead of recomputing the full prefix.
The unquantized size is 2 times layers times KV heads times head dimension times cached tokens times bytes per element times batch size.
Fanout's KV cache formula guide derives that expression and explains why grouped-query attention changes the KV-head term.
KV cache quantization changes the bytes used for those stored elements. It does not shrink model weights, reduce the number of cached tokens, or change the number of layers.
It also differs from PagedAttention. PagedAttention reduces allocation waste by placing cache entries in blocks. Quantization reduces the representation size inside those blocks.
The two methods can work together. One manages where the cache lives, while the other changes how many bits each cached value needs.
How much memory KV cache quantization saves
Use the published Llama 3.1 8B configuration: 32 layers, 8 KV heads, 32 query heads, and hidden size 4,096.
The head dimension is 4,096 divided by 32, which equals 128.
At BF16, one cached token for one sequence needs 2 times 32 times 8 times 128 times 2 bytes. That is 131,072 bytes, or 128 KiB.
At 128K tokens, the raw cache is 128 KiB times 131,072 tokens. The result is 16 GiB for one sequence.
FP8 uses one byte per stored value, so the same raw cache is 8 GiB. Four-bit storage uses half a byte per value, so its packed tensor is 4 GiB.
This arithmetic assumes every layer uses the same format and omits metadata. It also assumes the runtime packs four-bit values without padding that changes the total.
The headline compression is therefore 2 times for FP8 and 4 times for four-bit storage. Actual allocated memory is usually a little larger.
Scale metadata makes four bits cost more
Low-bit values need a rule that maps a small integer range back to the original numerical range. Group-wise affine quantization usually stores a scale and often a zero point for each group.
The Hugging Face implementation guide shows the scale and zero-point calculation used by its quantized cache and exposes group size as a configuration choice.
Consider an illustrative four-bit group with 64 values. The packed values take 64 times 4 bits, or 32 bytes.
If the group also stores one 16-bit scale and one 16-bit zero point, metadata adds 4 bytes. The group now takes 36 bytes.
That is 4.5 effective bits per value, not 4. Compared with 16-bit storage, the compression ratio is 16 divided by 4.5, or about 3.56 times.
This is an accounting example, not a universal format. Some schemes use symmetric quantization without a zero point. Others change scale precision, group size, alignment, or layout.
Smaller groups adapt better to local value ranges but need more metadata. Larger groups reduce metadata but force more values to share one scale.
Measure the runtime's allocated blocks rather than multiplying the nominal bit width and stopping there.
Residual tokens reduce the average compression
Quantizing every new key and value immediately can add overhead and amplify error near the current decoding position.
Hugging Face keeps a residual cache in the original precision and quantizes it when the residual reaches its limit. Its guide uses 128 tokens as a baseline residual length.
Return to the Llama 3.1 8B example. At four bits, a token's packed KV data is 32 KiB. At BF16, it is 128 KiB.
For a 128K-token sequence with 128 residual tokens, the ideal packed total is about 4.012 GiB before scale metadata. The residual window adds only 12 MiB over an all-four-bit cache.
At a 512-token context, the same residual has a much larger effect. The 384 quantized tokens use 12 MiB, while 128 BF16 tokens use 16 MiB.
The total is 28 MiB before metadata, compared with 64 MiB for a fully BF16 cache. That is about 2.29 times compression, not 4 times.
Residual length barely moves a very long context but can dominate a short one. This is one reason small prompts may show modest memory savings and worse latency.
Keys and values do not quantize the same way
One scale for an entire cache is cheap, but activation distributions are not uniform across tokens and channels.
The KIVI paper found persistent outlier channels in keys and used per-channel quantization for the key cache. Values were more suitable for per-token quantization.
KIVI kept an unquantized residual and quantized completed groups. Its two-bit method reported 2.6 times lower peak memory including weights, up to 4 times larger batches, and higher throughput in its tested workloads.
Notice the difference between a two-bit cache and 2.6 times lower total peak memory. Model weights, temporary buffers, metadata, and residual values do not become two-bit KV entries.
The KVQuant paper also used per-channel key quantization, plus pre-RoPE keys, non-uniform datatypes, and separate outlier handling for sub-four-bit precision.
These designs exist because simply rounding every cache value with one global scale can damage attention scores. Lower bit width makes the choice of grouping and calibration more important.
Smaller memory does not guarantee faster decode
Decode repeatedly reads the growing cache, so fewer bytes can reduce memory-bandwidth pressure. It can also allow more live sequences before the memory pool fills.
The gain depends on the kernel. A runtime must unpack or dequantize the cache before or during attention, and that work can cost more than the saved transfer time at short contexts or low concurrency.
KIVI used custom hardware-friendly kernels for its throughput results. Those numbers do not transfer automatically to a generic cache implementation.
NVIDIA's NVFP4 KV cache results report lower capacity and bandwidth use on Blackwell hardware.
That implementation dequantizes four-bit cache data to FP8 for attention and depends on NVIDIA's kernels and formats. Treat the reported gains as evidence for that stack, not a promise for every GPU.
Quantization can improve throughput without reducing latency for one request. Extra capacity may let the scheduler batch more sequences, while a lone short sequence still pays conversion overhead.
The prefill and decode guide explains why the same optimization can affect the two phases differently.
What current runtimes support
Current vLLM quantized-cache documentation supports FP8 cache formats and several scale-calibration paths.
It distinguishes per-tensor scales from per-attention-head scales and can skip selected layers that are more sensitive to cache quantization.
Current Transformers cache documentation lists HQQ support for 2, 4, and 8 bits, while Quanto supports 2 and 4 bits.
Bit width alone does not describe runtime support. Check the attention backend, GPU generation, data type, calibration path, group size, residual length, and whether every layer is quantized.
Also verify that your serving engine reserves a larger token pool after the change. A smaller tensor is useful only if the allocator and scheduler can use the freed capacity.
Measure memory, quality, and latency together
Start with the byte formula for your model, then add the format's scale metadata, residual cache, unquantized layers, and block-allocation behavior.
Run the same prompt and generation traces with the baseline cache and each candidate format. Keep model revision, sampling settings, batch policy, and attention backend fixed.
Record at least:
- Allocated KV-cache bytes at several context lengths.
- Maximum live tokens and sequences before preemption or out-of-memory failure.
- Time to first token and inter-token latency by context bucket.
- Input and output tokens per second under the target concurrency.
- Task quality on long-context retrieval and the actual application workload.
- Quantization and dequantization time inside the attention path.
FP8 is often the simpler first test because its raw compression is 2 times and hardware support is broader. Four bits can release more capacity, but metadata, residuals, quality, and kernels matter more.
KV cache quantization is worthwhile when cache capacity or bandwidth is the measured limit. The real saving is the new concurrency or context that fits while quality and latency remain inside the service target.