NVFP4 KV cache vs NVFP4 weights

NVFP4 KV cache vs NVFP4 weights is not a choice between two names for the same quantization. The weights are fixed model parameters. The cache is runtime state that grows with every active token.

Both can store E2M1 values with block scales, but they use different kernels and solve different memory limits. A server can use NVFP4 weights with FP8 cache, or BF16 weights with NVFP4 cache.

Choose weight precision from model fit and matrix multiplication. Choose cache precision from context, concurrency, attention bandwidth, and measured quality.

NVFP4 KV cache vs NVFP4 weights

NVIDIA's NVFP4 format description uses 4-bit E2M1 values and an E4M3 scale for each block of 16 values.

The small block limits the range shared by unrelated values. A second scale can map the tensor into the range represented by the FP4 values and their block scales.

That numerical format does not decide what the values mean. A block may hold model weights, temporary activations, keys, or values.

Weights exist before a request arrives and remain read-only during inference. The KV cache is created during prefill, appended during decode, read by attention, and freed or evicted later.

Those lifecycles determine calibration, memory growth, and kernel cost.

Fanout's FP8 attention vs FP8 KV cache guide applies the same separation to an 8-bit format.

NVFP4 weights shrink a fixed model

Weight quantization converts large parameter matrices before serving or during model load. The quantized checkpoint stores the codes and their scale metadata.

Native Blackwell kernels can consume NVFP4 weights in block-scaled matrix multiplication. Depending on the serving path, activations may also be FP4 or may stay at a wider type.

The memory saving is mostly fixed per replica. It does not grow when a request adds another token.

Ignoring global scales, padding, and unquantized layers, each group of 16 weights needs 64 data bits plus one 8-bit block scale.

That is 72 bits for 16 values, or 4.5 bits per weight. A 70-billion-parameter toy model would need about 39.4 GB of payload instead of 140 GB for pure BF16 weights.

Real checkpoints are larger than this simple payload. Embeddings, norms, biases, global scales, alignment, and excluded layers may use wider storage.

The useful comparison comes from the actual checkpoint and loaded GPU allocation, not parameter count times four bits.

NVFP4 cache shrinks growing runtime state

For a standard attention cache, values per token equal 2 times layers times KV heads times head dimension. The factor 2 accounts for keys and values.

Take an 80-layer model with 8 KV heads of width 128. One token stores 2 times 80 times 8 times 128, or 163,840 values.

At BF16, that is 327,680 bytes, or 320 KiB per token. FP8 needs 160 KiB before any separate scale metadata.

At an idealized 4.5 bits per value, NVFP4 needs 92,160 bytes, or 90 KiB per token. One 32,768-token sequence then holds about 2.81 GiB of cache.

The same sequence needs 10 GiB in BF16 or 5 GiB in plain one-byte FP8. Eight such sequences need about 22.5 GiB in idealized NVFP4 instead of 80 GiB in BF16.

This saving scales with live tokens. It matters little at short contexts and low concurrency, then dominates when the cache pool limits admitted work.

The KV-cache memory formula covers grouped-query attention, tensor parallelism, and the dimensions hidden by this toy calculation.

The cache quantizes values during serving

Weights can be calibrated against representative data and then frozen. The KV cache must quantize new key and value vectors produced by each request.

Current vLLM NVFP4 cache code packs FP4 data and writes FP8 block scales into paged cache slots.

The attention backend later reads the packed values and scales. It may dequantize them or feed a compatible low-precision path, depending on GPU architecture and kernel.

The store path therefore pays for scale calculation, rounding, packing, and slot mapping while requests are running.

The read path pays for unpacking or low-precision attention support. Weight GEMM speed does not prove that the attention kernel handles NVFP4 cache efficiently.

Scale quality also has a different input distribution. Weight ranges are fixed, while keys and values depend on prompts, layers, positions, and the current model.

Validate the cache on the same long-context and reasoning workloads the server will carry.

The two controls are independent in vLLM

The completed vLLM NVFP4 cache issue shows the cache enabled with the runtime flag --kv-cache-dtype nvfp4.

Weight format comes from the model checkpoint and its quantization configuration, or from a separate quantization choice such as modelopt_fp4.

An example in that issue combines an NVFP4 weight checkpoint with NVFP4 cache. Changing only the cache flag does not rewrite the checkpoint's weights.

The current FlashInfer backend lists NVFP4 among supported cache dtypes. It also has layout constraints, including equal key and value head sizes on the cited path.

Support must therefore be checked as a matrix: GPU architecture, model weight format, attention backend, cache dtype, head layout, and framework version.

A model loading successfully proves only the weight path. Allocate the cache, run prefill and decode, and inspect which attention kernels actually execute.

Four bits do not guarantee a speedup

NVIDIA's NVFP4 KV cache study reports about half the cache memory of FP8.

It also reports less than 1 percent accuracy loss on its tested coding, knowledge, and long-context benchmarks. Those results are tied to the named models, kernels, and workloads.

A recent vLLM performance report shows why capacity and speed need separate measurements.

On one SM120 setup, 48 matched causal-prefill calls took 145.8 ms with NVFP4 cache and 84.0 ms with FP8, about 1.74 times slower.

The same report measured decode about 22 percent faster with NVFP4 in that workload. Decode reads a growing cache and is more likely to benefit from lower bandwidth.

Its append kernel was about 2.8 times slower, though the reported cumulative append time was only about 7 ms.

With 30 concurrent requests, NVFP4 completed without preemption but took 102.3 seconds. FP8 preempted six requests yet finished in 65.4 seconds.

This is one engineering report, not a universal benchmark. It demonstrates the tradeoff: smaller cache can admit more work while conversion overhead still hurts a prefill-heavy run.

Quantize the memory pool that is binding

If model weights do not fit, weight quantization solves the first admission problem. It can also reduce weight bandwidth and accelerate supported GEMMs.

If the model fits but long contexts or concurrency exhaust the cache pool, KV quantization addresses the growing part of memory.

If decode is bandwidth-bound, the smaller cache may help latency or throughput. If prefill dominates, dequantization and quantized-store costs may erase that benefit.

Combining both can free the largest total amount of memory, but it also stacks two separate sources of quantization error.

Run four controlled configurations when the backend permits them: wide weights with wide cache, NVFP4 weights only, NVFP4 cache only, and both.

Keep model, requests, scheduler, prefix-cache state, and memory utilization fixed. Otherwise a larger cache may change concurrency or hit rate and obscure the precision effect.

Measure weights and cache on separate axes

For weights, record checkpoint bytes, loaded GPU memory, prefill throughput, decode throughput, and task quality.

For cache, record bytes per token, cache capacity, admitted sequences, preemptions, hit rate, time to first token, inter-token latency, and long-context quality.

Confirm the actual cache allocation rather than assuming four bits means exactly one quarter of BF16. FP8 scales, alignment, page metadata, and backend padding consume space.

Profile prefill and decode separately. Their arithmetic intensity and cache access patterns differ, so an average request latency can hide the slower phase.

Keep the wide-cache result even after NVFP4 weights pass quality checks. Keep the wide-weight result when testing NVFP4 cache. Those controls identify which quantization caused a regression.

NVFP4 weights reduce a mostly fixed model footprint. NVFP4 KV cache reduces state that grows with tokens and users. The shared number format does not make their benefits, risks, or kernels interchangeable.