--- title: "W4A8 vs NVFP4 quantization on GPUs" description: "Compare integer W4A8 with NVFP4 by number format, scale overhead, GPU support, kernel behavior, and model quality before choosing a serving path." canonical_url: "https://fanout.sh/blog/w4a8-vs-nvfp4-quantization" md_url: "https://fanout.sh/blog/w4a8-vs-nvfp4-quantization.md" last_updated: "2026-08-31" access: "public" --- # W4A8 vs NVFP4 quantization on GPUs Compare integer W4A8 with NVFP4 by number format, scale overhead, GPU support, kernel behavior, and model quality before choosing a serving path. - Author: Suraj Gaud - Published: 2026-08-31 - Track: Inference engineering - Access: Public - Tags: W4A8, NVFP4, quantization, LLM inference, Blackwell, INT4, FP4, TensorRT-LLM W4A8 vs NVFP4 quantization is not a contest between two interchangeable 4-bit checkpoints. W4A8 names the bit widths of weights and activations. NVFP4 specifies a number format, scale layout, and native Blackwell compute path. Choose from the GPU backward. On Hopper, integer W4A8 has supported kernels while NVFP4 does not have native FP4 Tensor Cores. On Blackwell, NVFP4 usually deserves the first benchmark because the hardware consumes its FP4 layout directly. Model quality and batch shape can reverse that preference. The useful comparison is a four-part one: stored values, scale metadata, kernel work, and accuracy on the requests you serve. ## W4A8 vs NVFP4 quantization In an integer W4A8 path, weights use 4-bit integers and activations use 8-bit integers. A scale maps each integer code back toward the original tensor value before or during matrix multiplication. The exact recipe still matters. W4A8 can mean AWQ, GPTQ, QServe, per-channel scales, or grouped scales. Two checkpoints with the W4A8 label can require different layouts and kernels. [QServe](https://arxiv.org/abs/2405.04532)uses W4A8KV4: 4-bit weights, 8-bit activations, and a 4-bit KV cache. Its algorithm and GPU kernels were designed together to keep dequantization off slow execution paths. NVFP4 uses E2M1 values. Each four-bit code has one sign bit, two exponent bits, and one mantissa bit. The representable magnitudes are unevenly spaced. That fits long-tailed weight and activation distributions better than a uniform integer grid. NVFP4 normally quantizes both weights and activations to FP4. That makes it closer to W4A4 in bit width, but the native floating-point representation makes the operational comparison with W4A8 useful. Fanout's[W4A8 QServe guide](/blog/w4a8-quantization-qserve)covers the integer path in more detail. The[W8A8 vs W4A16 comparison](/blog/w8a8-vs-w4a16-quantization)explains why bit labels alone do not predict speed. ## Scale layout changes the real storage cost Four bits per weight is only the payload. Quantized tensors also carry scales and sometimes zero points, so group size changes the effective bits per weight. Suppose an INT4 weight tensor stores one FP16 scale for every 128 weights. The scale adds 16 / 128 = 0.125 bits per weight. The payload plus that scale costs 4.125 bits per weight before alignment or zero-point overhead. Change the group to 32 and the same FP16 scale adds 0.5 bits per weight. Finer groups can reduce error, but they consume more metadata and create more scale work in the kernel. NVFP4 fixes the local group at 16 values and stores one E4M3 FP8 scale per group. That adds 8 / 16 = 0.5 bits, producing 4.5 bits per value, plus one FP32 scale for the whole tensor. The[NVIDIA NVFP4 description](https://developer.nvidia.com/blog/introducing-nvfp4-for-efficient-and-accurate-low-precision-inference/)explains the two levels. The FP8 local scale fits each 16-value block, while the tensor scale restores range. The local E4M3 scale can represent fractional values. MXFP4 instead uses an E8M0 power-of-two scale for 32 values. This distinction matters because NVFP4 is not simply INT4 with a different name. ## Hardware support is the first filter The current[TensorRT-LLM support matrix](https://nvidia.github.io/TensorRT-LLM/features/quantization.html)lists NVFP4 and MXFP4 on Blackwell, not Hopper. It lists W4A8 AWQ and GPTQ on Hopper and some earlier GPUs. A TensorRT-LLM issue about[W4A8 FP4 checkpoints on Hopper](https://github.com/NVIDIA/TensorRT-LLM/issues/12558)shows the failure mode. The checkpoint can load far enough to reach a kernel dispatch, then fail because the FP4 GEMM architecture is unsupported. An NVIDIA collaborator confirmed in that issue that NVFP4 and MXFP4 are not supported on Hopper. A weight-only or hybrid name does not create FP4 Tensor Cores where the architecture lacks them. This gives a simple first branch. Use an integer or FP8 route on A100, L40S, H100, and H200. Put NVFP4 on the shortlist for B200, B300, GB200, GB300, and supported Blackwell client GPUs. Framework support remains narrower than architecture support. Check the exact model family, backend, and quantization row instead of reading a single green cell as universal support. ## Dequantization can erase a four-bit win An INT4 matrix unit cannot consume every W4A8 checkpoint layout directly. Kernels may unpack nibbles, load scales, convert values, reorder weights, or accumulate partial sums at a higher precision. QServe reports that earlier INT4 methods spent 20% to 90% of runtime on weight or partial-sum dequantization. That overhead explains why a smaller checkpoint can run slower than a wider format with a better hardware path. QServe avoids a generic unpack-then-multiply sequence. Progressive quantization, weight reordering, and register-level parallelism move the conversion work into a layout built for its W4A8 GEMM. Its paper measured higher maximum serving throughput than TensorRT-LLM on the tested A100 and L40S systems. Those results validate QServe's complete stack, not every file labeled W4A8. Blackwell Tensor Cores implement NVFP4 scaling and FP4 matrix operations in hardware. The kernel still has scale traffic and accumulation work, but it does not need to emulate the FP4 operation through an unrelated integer path. The checkpoint that wins is therefore the checkpoint whose layout reaches a mature native kernel. File size cannot reveal that. ## Batch size can change the winner At a small batch, decode often spends much of its time reading weights. Cutting weight bytes can help even when some unpacking remains, because memory traffic dominates. At a larger batch, matrix multiplication does more work per weight load. Arithmetic throughput and conversion instructions matter more, so dequantization overhead can become visible. QServe was built around this problem. Its authors note that many earlier INT4 techniques accelerated low-batch inference but failed to improve large-batch cloud serving. NVFP4 attacks the same boundary differently. On Blackwell, the 4-bit floating-point operation is a native Tensor Core mode, so the format has a stronger path when the workload becomes compute-bound. Do not benchmark one batch and extrapolate. Sweep request rate, prompt length, output length, and concurrency. Record TTFT, inter-token latency, output throughput, and SLO-qualified goodput for each point. Fanout's[goodput vs throughput guide](/blog/goodput-vs-throughput-llm-inference)explains why the highest token rate may still be the wrong production configuration. ## Quality depends on format and calibration NVFP4's nonuniform E2M1 values give more codes to small magnitudes while retaining values up to 6 in each scaled block. INT4 places codes on a uniform grid after scaling. A[controlled quantization evaluation](https://arxiv.org/html/2507.17417v2)compared INT4 and FP4 at matched granularities. On Llama 3.2 1B, GPTQ gave WikiText-2 perplexity 11.47 for INT4 group 16 and 10.98 for NVFP4-style group 16. That experiment isolates format behavior. It is not proof that NVFP4 serving is faster, and one small model's perplexity is not a production acceptance test. The same study found that scale format and group size materially affected FP4 quality. Its NVFP4 layout used 16-value groups, E4M3 local scales, and a second FP32 tensor scale. NVIDIA reports small benchmark changes for several large NVFP4 models, while Red Hat found more variability among 7B to 14B models. Both sets of results point toward model-specific evaluation rather than a universal quality promise. Use calibration data that resembles production prompts. Compare task accuracy, long-context behavior, structured output validity, and refusal behavior against the higher-precision checkpoint. ## A practical selection sequence Start with the deployed GPU. If it lacks native FP4 support, remove NVFP4 from the performance shortlist and benchmark supported W4A8, W4A16, W8A8, and FP8 recipes. On Blackwell, check whether the exact model and backend have an NVFP4 kernel. A pre-quantized checkpoint with a supported path is a better starting point than a hand-converted file with uncertain metadata. Next, calculate effective storage, including scales. Then run a batch sweep instead of comparing one headline throughput number. Finally, place the quality gate beside the latency gate. Reject a format if it wins tokens per second but misses the task-level accuracy or SLO required by the service. W4A8 remains useful because it works across more GPU generations and mature integer kernels can control conversion overhead. NVFP4 is compelling on Blackwell because the format, scale layout, and Tensor Core path were designed together. Choose the complete checkpoint-to-kernel path that produces the most correct, SLO-qualified output on the hardware you own. --- This representation contains public Fanout content only. Protected Pro lessons, account data, billing, checkout, and pricing are not included. Browse the public content map: https://fanout.sh/sitemap.md