NVFP4 KV cache A4Q prefill explained

NVFP4 KV cache A4Q prefill targets an awkward result on consumer Blackwell: a cache that is smaller and faster for decode can still make causal prefill slower than FP8.

The stock path reads packed FP4 keys and values, expands them, applies block scales, and performs attention with BF16 queries. A4Q instead quantizes each query tile and uses native block-scaled FP4 matrix multiplication for QK.

That removes one conversion path, but it does not make the whole attention calculation FP4. It also does not erase the need to measure query quantization error.

Why NVFP4 KV cache A4Q prefill exists

NVFP4 stores two E2M1 values per byte plus an FP8 scale for each 16-value block. The layout cuts cache bytes, but a BF16-oriented attention kernel must unpack and scale K before QK.

The reported SM120 path also dequantizes V for the probability-times-value calculation. Those conversions happen inside a compute-heavy causal-prefill kernel.

Fanout's NVFP4 cache and weight comparison explains why native FP4 weight GEMM does not prove that the cache path is fast.

An open vLLM performance report compared 48 matched causal paged-prefill calls on an RTX PRO 5000 Blackwell.

NVFP4 took 145.81 ms and FP8 took 84.03 ms. The ratio was 1.74, while CUTLASS GEMM time elsewhere in the same segment stayed similar.

The report attributes the extra time to packed FP4 unpacking, conversion, and scale application inside attention. It labels that explanation a hypothesis rather than an instruction-level proof.

A4Q changes the QK path. It quantizes Q to a compatible block-scaled FP4 representation, then consumes packed K and its cache scales in native FP4 MMA.

The proposed path keeps the output and the PV calculation wider. The V scale axis does not line up with the reduction axis in the same convenient way.

Work the stock prefill measurements

The matched 48-call segment averaged 3.04 ms per NVFP4 call and 1.75 ms per FP8 call. That is the same 1.74 ratio as the totals.

Decode moved in the other direction. The report measured about 9.63 microseconds per NVFP4 decode call and 12.35 microseconds for FP8, making NVFP4 about 22 percent faster there.

KV append cost about 8.39 microseconds with NVFP4 and 3.01 microseconds with FP8. Packing and scale generation made append about 2.8 times slower, but its cumulative time was only about 7 ms.

The end-to-end stress test admitted all 30 requests with NVFP4 and recorded no preemptions. FP8 peaked at 24 running requests and preempted six.

NVFP4 still finished in 102.3 seconds versus 65.4 seconds for FP8. In that prefill-heavy workload, capacity prevented preemption but did not repay the slower attention kernel.

The cache footprint was 56.25 percent of the FP8 payload in the report. Its ideal capacity multiplier is 1 divided by 0.5625, or about 1.78.

The server can hold more tokens, yet each cold or missed prefix can cost more to ingest.

NVIDIA's NVFP4 KV cache study reports larger-context and hit-rate gains on different models and software.

Do not combine the two studies into one benchmark. One isolates an SM120 prefill regression, while the other measures co-designed capacity and caching outcomes on its named stack.

Quantize Q instead of expanding K

The issue thread reports a controlled 16K-token, single-request run. Stock NVFP4 averaged 21.0 ms per prefill launch, while FP8 averaged 11.1 ms.

That makes the stock NVFP4 path 1.89 times the FP8 time for this shape. The larger ratio than the earlier 1.74 result came from a longer prompt and a different controlled slice.

An out-of-tree A4Q-style kernel averaged 18.0 ms on the same protocol. Dividing 18.0 by 21.0 gives 0.857, so the native FP4 QK path removed about 14.3 percent of stock NVFP4 kernel time.

It did not catch FP8. Dividing 18.0 by 11.1 gives about 1.62, leaving a 62 percent kernel gap on that full-prefill shape.

At 32K tokens, the same report says TTFT moved from 1.73 times FP8 to 1.52 times FP8. That is evidence from an experimental branch, not an upstream FlashInfer guarantee.

The A4Q kernel consumed K directly as packed E2M1 values with the cache's own block scales. It quantized Q inside the kernel and kept PV in BF16.

Quantizing Q adds its own work. The path wins only if the native FP4 QK saving exceeds query scale calculation, packing, and any accuracy guardrails.

Compare A4Q with dequantize-ahead repacking

Another optimization converts an NVFP4 tile once before repeated MMA use. It still produces BF16 or FP16 K and V, but avoids repeating unpack and scale work inside every MMA iteration.

The open FlashInfer repack PR reports a causal paged-prefill benchmark with batch 4, query length 128, KV length 2,048, and head dimension 128.

Its parent measured 0.2999 ms after warmup. The repack path measured 0.2888 ms, a 3.7 percent reduction. A shorter 256-token KV case stayed within noise.

That result is narrower than the full-prefill A4Q report. It shows one long-KV, short-query case, not a repair for every causal-prefill shape.

The two techniques attack different repetition. Repacking amortizes K and V conversion when a tile is consumed across several MMA iterations.

A4Q avoids widening K for QK and uses Blackwell's block-scaled FP4 MMA. It still needs a wider route for PV in the reported design.

Full prefill has query length near KV length and a large QK workload. A native FP4 QK path has more arithmetic to influence there.

Chunked-prefill tails can have 8 to 16 query tokens attending to roughly 16K cached tokens. Reusing a converted KV tile may matter more in that short-Q, long-KV regime.

Treat repacking and A4Q as complementary candidates until the same benchmark matrix tests both. Comparing numbers from different shapes cannot select a winner.

Preserve the capacity and decode case

The FlashInfer FP4 utility work provides GPU quantization and dequantization kernels with a linear block-scale layout.

That plumbing is useful even before a native FP4 attention path wins every phase. Cache allocation, writes, transfers, and fallbacks still need a defined representation.

Fanout's KV-cache memory formula supplies the uncompressed bytes per token. Multiply that value by the measured NVFP4-to-reference storage ratio for a capacity estimate.

Keep prefill and decode on separate rows. Prefill can be compute-heavy and conversion-sensitive, while decode reads a long cache for one new query token and is often bandwidth-bound.

Also separate cold prefill from prefix hits. A smaller cache can retain more prefixes, so a slower miss path may still improve workload TTFT if enough requests avoid prefill entirely.

The break-even condition uses request mix. Extra time on cold and missed prompt tokens must be smaller than time saved by extra cache hits, fewer preemptions, and faster decode.

Do not use cache occupancy alone. The 30-request test shows that admitting more work can still increase wall time when the expensive phase dominates.

Make query accuracy a release gate

Native FP4 QK dynamically quantizes Q, which changes the attention scores before softmax. Long contexts can amplify small score changes when many keys compete.

Test attention outputs against a wide reference across prompt lengths, head shapes, causal masks, and scale outliers. Kernel speed without numerical coverage is incomplete.

Then run model-level long-context, retrieval, code, and reasoning evaluations. Use the same calibrated cache and exact kernel revision planned for serving.

Record whether Q uses one scale or a dual-level scheme, the 16-value grouping, rounding mode, and any residual scale. A label such as A4Q does not identify those choices.

Check output stability at temperature zero and distribution shifts at sampled temperatures. A passing short perplexity test does not cover long-context retrieval.

The A4Q report itself calls the numerical impact an open concern. Keep that uncertainty visible rather than turning a performance prototype into a quality claim.

Build the benchmark matrix

Use at least four query shapes: full 8K prefill, full 32K prefill, a 128-token query over 2K KV, and an 8 to 16-token tail over 16K KV.

Run FP8, stock NVFP4, NVFP4 repack, and A4Q-style NVFP4 when those paths are available on the same revision.

For each case, record kernel time, TTFT, decode time, append time, cache bytes per token, preemptions, peak running requests, and task quality.

Capture kernel names and dtypes from the trace. A server flag does not prove that the optimized dispatch ran instead of a fallback.

Keep batch, page size, layout, heads, head dimension, prefix-cache warmth, and scheduler limits fixed. One changed geometry can move the result more than the precision choice.

Choose the read path by whether it repays conversion and query quantization for the service's actual prompt shapes, not by a general claim that NVFP4 is fast.

NVFP4 KV cache A4Q prefill can remove part of the stock SM120 penalty. The measured prototype narrowed the gap, while repacking helped a different shape. Neither result replaces a workload-level break-even test.