When KV cache quantization slows inference
KV cache quantization halves or quarters the bytes each cached token needs, yet turning it on can make a server measurably slower. Both observations are correct, and the difference is mechanical, not mysterious.
When KV cache quantization slows inference, one of four factors is responsible: the attention kernel reading the cache, conversion work on the decode path, context length, or freed memory that never becomes batched work.
Each factor has numbers attached. Walk through them before flipping the flag in production.
The speedup never comes from quantization itself
Quantizing the cache shrinks stored keys and values. It does not make attention arithmetic faster on its own, and it adds encode and decode work around every cache access.
The wins arrive indirectly. Smaller entries mean less memory traffic per decode step, and more free memory means the scheduler can keep more sequences alive at once.
A benchmark by SqueezeBits compared the two paths on the same model and GPU across vLLM and TensorRT-LLM.
Its throughput gains traced to the batching effect: quantized caches let more requests into each iteration.
If neither bandwidth nor capacity is your binding constraint, there is no channel through which quantization can pay. The conversion overhead remains, so the net effect is a slowdown.
Fanout's KV cache quantization guide works through the memory-side accounting: scale metadata, residual tokens, and why nominal ratios overstate savings.
The attention backend decides the kernel path
The same quantization flag produced opposite results in the SqueezeBits test, and the backend explains it.
On vLLM v0.6.3, FlashAttention-2 did not support an FP8 cache, so enabling it forced a fall back to slower attention implementations. FP8 gave minimal or negative throughput impact, with slight degradation in the prefill-heavy scenario.
TensorRT-LLM v0.13.0 had FP8 attention kernels ready. The same model and GPU gained up to 1.09 times in the prefill-heavy scenario and 1.45 times in the decode-heavy one.
The lesson is that a cache format is only as fast as the kernel that reads it. An engine without a fused path must dequantize before attention, and that work can erase the bandwidth saved.
Backend support keeps shifting. Current vLLM quantized cache documentation has FlashAttention-3 run attention in the FP8 domain and quantize queries to match.
Check the backend your engine actually selects with the flag enabled. A silent fallback is the most common way this feature loses speed.
Conversion work rides the decode critical path
Every new token's key and value must be quantized before storage, and stored entries must be interpreted at every subsequent step.
Fanout's prefill and decode guide explains why per-step costs multiply across a generation.
The Hugging Face quantized cache implementation keeps a residual buffer of recent tokens in original precision and quantizes it when it fills. The default residual length is 128 tokens.
Even with that design, its authors measured slower generation at larger batch sizes with an INT4 cache, and they frame the tradeoff as speed given up in exchange for memory.
Stacking methods compounds the cost. The same post reports that weight quantization combined with cache quantization can produce a threefold decrease in generation speed.
Lower bit widths buy more capacity but demand more work per access: grouped scales, zero points, and unpacking. FP8 is the cheapest conversion, which is why engines support it first.
Short contexts pay the cost without the benefit
The bandwidth saving scales with cached tokens, but the overhead per step is roughly fixed. Short sequences therefore sit on the losing side of the ratio.
The residual buffer makes this concrete. With 128 full-precision residual tokens, a 512-token context holds a quarter of its cache unquantized, and measured compression drops to near 2.3 times instead of 4.
At 128K tokens the same residual is noise, and the KV cache memory formula says the cache dominates GPU memory.
That is where an 80 GB A100 goes from roughly 40K to 128K cacheable tokens with quantization.
A chat workload with 1K-token conversations gains little capacity, saves little bandwidth, and still pays conversion on every step. A long-document workload flips every one of those terms.
Estimate cache bytes at your real context distribution before benchmarking. If the unquantized cache is a small fraction of memory, quantization has nothing to free.
Throughput can rise while each request gets slower
Freed cache capacity raises throughput by admitting more concurrent sequences. Each individual request can still see worse inter-token latency, because conversion overhead applies per step and batches are now larger.
A server at low traffic never cashes in the capacity. The batch was small before quantization, it stays small after, and the only change is added kernel work.
This is why single-stream benchmarks and saturated benchmarks disagree about the same configuration. The SqueezeBits decode-heavy gain of 1.45 times appeared under load, where extra batch slots turned into served requests.
Accuracy rarely settles the question at 8 bits: the same test measured MMLU within a fraction of a point of the BF16 baseline on both engines. Speed and capacity, not quality, decide the FP8 case.
Decide what you are optimizing first. A latency target argues for testing quantization off; a cost-per-token target usually argues for it, once the kernels are confirmed.
Engine versions and hardware move the answer
The SqueezeBits numbers date to vLLM v0.6.3 and TensorRT-LLM v0.13.0 on an H100 PCIe. Kernel support has widened since, and results from one version do not transfer.
vLLM today documents per-attention-head FP8 scales, calibration through llm-compressor, and a flag to skip layers that are sensitive to cache quantization, such as sliding-window attention.
NVIDIA's NVFP4 cache stores 4-bit entries and dequantizes to FP8 for attention on Blackwell.
Older GPUs lack that path. Research formats push further down: KIVI reaches 2 bits with per-channel key quantization, but its throughput results ride on custom kernels a stock engine may not have.
So retest after every engine upgrade. A configuration that lost speed last quarter may win now, and the reverse happens when a backend rewrite drops a fused path.
When KV cache quantization slows inference, in short
KV cache quantization slows inference when the attention backend lacks a fused path, when contexts are short, when concurrency is too low to use freed capacity, or when stacked quantization overloads the decode path.
It pays when long contexts make cache bandwidth and capacity the binding constraint and the kernels exist to read the format directly.
The only trustworthy verdict comes from your engine version, your GPU, and your traffic, with the attention backend logged for both runs.
- Confirm which attention implementation loads with the quantized cache enabled.
- Benchmark at your real context-length distribution, not one synthetic length.
- Test at production concurrency and at single-stream, since the verdicts differ.
- Record time to first token, inter-token latency, and throughput together.
- Check allocated cache blocks to verify freed memory became scheduled work.
The flag is cheap to test. The mistake is trusting a result measured on someone else's stack.