--- title: "W4A8 quantization in QServe" description: "See why QServe stores 4-bit weights, computes on INT8 tensor cores, and pairs W4A8 with a 4-bit KV cache for batched LLM serving." canonical_url: "https://fanout.sh/blog/w4a8-quantization-qserve" md_url: "https://fanout.sh/blog/w4a8-quantization-qserve.md" last_updated: "2026-08-24" access: "public" --- # W4A8 quantization in QServe See why QServe stores 4-bit weights, computes on INT8 tensor cores, and pairs W4A8 with a 4-bit KV cache for batched LLM serving. - Author: Suraj Gaud - Published: 2026-08-24 - Track: Inference engineering - Access: Public - Tags: W4A8, QServe, quantization, LLM inference, INT4, INT8, KV cache, GPU kernels, QoQ W4A8 quantization in QServe does not run every operation as a native 4-bit calculation. It stores weights in 4 bits, turns them into an 8-bit intermediate inside the GEMM kernel, and multiplies on INT8 tensor cores. That distinction explains why QServe can beat W4A16 and W8A8 in its serving benchmarks. Four-bit storage saves weight bandwidth, while 8-bit activation compute avoids the expensive partial-sum path that hurt earlier W4A4 systems. QServe actually uses W4A8KV4: 4-bit weights, 8-bit activations, and a 4-bit KV cache. Each precision choice targets a different bottleneck. ## W4A8 quantization in QServe The notation WxAy states the stored weight precision and activation precision. W4A8 means 4-bit weights and 8-bit activations for matrix multiplications. For an 8-billion-parameter dense model, raw weight storage is about 4 GB at 4 bits per parameter. The same count is about 8 GB at 8 bits and 16 GB at 16 bits, before scales, metadata, and padding. Weight-only W4A16 also gets the 4 GB starting point. Its problem is compute: the kernel must convert INT4 weights to floating point before multiplying them with FP16 activations. W8A8 uses twice as many bytes for weights, but the matrix multiply maps cleanly to fast INT8 tensor cores. That becomes attractive when a large batch makes the operation compute-bound. The[QServe paper](https://arxiv.org/abs/2405.04532)frames W4A8 as the bridge between those regimes. It keeps W4 weight traffic and uses INT8 tensor-core arithmetic once the weights are unpacked. This is narrower than a general claim that W4A8 is always fastest. The result depends on a kernel that makes the INT4-to-INT8 conversion cheap enough. ## Why lower precision can run slower Bit width determines traffic and peak arithmetic rates, but neither number includes instructions around the tensor-core operation. The QServe authors measured 20 to 90 percent overhead in existing INT4 methods when they dequantized weights or partial sums on GPUs. That work runs on CUDA cores inside a sequential GEMM main loop. The paper estimates that one A100 CUDA-core operation can cost as much as 50 INT4 tensor-core operations. A nominally faster W4A4 multiply can therefore wait on conversion and accumulation code. W4A16 has a related cost. It loads compact weights, then converts them to FP16 in the loop so they can multiply FP16 activations. QServe instead unpacks four INT4 weights at once into an INT8 representation. It reorders static weights ahead of time so each thread can use wide memory transactions and fewer address calculations. The kernel then performs INT8 matrix multiplication. Per-channel FP16 scaling happens in the epilogue, outside the repeated main-loop work. This is why the[Fanout quantization comparison](/blog/fp8-vs-int8-vs-awq-vs-gptq)separates a numeric format from the kernel and hardware path that execute it. ## Progressive quantization makes W4A8 computable QServe's QoQ algorithm quantizes weights in two stages. It first maps each output channel to an INT8 intermediate, then compresses groups of those integers to unsigned INT4. At runtime, the kernel only needs to recover the INT8 intermediate. It does not reconstruct the original floating-point weight inside the main loop. There is a small overflow trap. Suppose an INT8 group spans -113 to 120. An asymmetric 4-bit scale of 16 and zero point of 7 map 120 to the top code, 15. Dequantizing that code gives (15 - 7) times 16, or 128. Signed INT8 stops at 127. Saturating the result sounds harmless, but the paper reports that enabling saturation cut throughput by as much as 67 percent in this path. QoQ prevents the overflow before it happens. It restricts the first quantization stage to the protective range from -119 to 119. The largest second-stage scale is 17. Leaving half a scale of headroom keeps the reconstructed value at or below 127 after rounding. That range is a system choice as much as a quantization choice. A slightly narrower representation makes the fused GPU implementation possible without a slow saturation instruction. ## Batch size changes the winning format During decoding, the GEMM's short dimension is the number of active sequences. Small batches tend to be limited by reading weights; large batches do enough reuse to become compute-bound. In the paper's A100 roofline model, W4A16 has the higher theoretical throughput below batch 78. W8A8 wins above 78 because INT8 tensor cores outrun FP16 tensor cores once compute becomes the limit. The proposed W4A8 roofline sits above both. It reads half as many weight bits as W8A8 and still performs the multiply on INT8 tensor cores. That roofline is a ceiling, not a benchmark. QServe still had to fuse activation quantization, reorder weights, reduce pointer arithmetic, and parallelize unpacking to approach it. The comparison also assumes the model and batch fit the device. Compact weights can leave more memory for active requests, but serving capacity also includes the KV cache and runtime workspace. The[W8A8 versus W4A16 guide](/blog/w8a8-vs-w4a16-quantization)is the useful baseline: W4A8 makes sense when a serving kernel can preserve the best part of each path rather than paying both conversion costs. ## Why QServe adds a 4-bit KV cache Weight precision controls much of the GEMM traffic. Decode attention has a different shape: every new query reads keys and values from earlier tokens. QServe labels its full format W4A8KV4 because it stores that growing cache in 4 bits. Compared with KV8, the raw cache traffic and storage are halved. Direct KV4 quantization damaged accuracy because key tensors had stable outlier channels roughly ten times larger than most values in the authors' samples. SmoothAttention moves those outliers from keys to queries through an inverse pair of channel scales. Keys become easier to quantize, while queries stay unquantized and preserve the attention dot product. The attention kernel still has to unpack the cache cheaply. QServe replaced FP32 work with FP16 operations and reduced dequantization to two operations per element to keep the kernel memory-bound. The authors report a 1.5 times speedup over TensorRT-LLM's KV8 kernel on A100 for that optimized path. The broader KV memory arithmetic is covered in Fanout's[KV cache formula walkthrough](/blog/kv-cache-memory-formula-llm-inference). QServe changes the bytes per element, not the number of cached layers, tokens, or KV heads. ## What the QServe benchmarks establish The MLSys 2025 paper evaluated seven model families on A100 and L40S GPUs. Its workload used 1,024 input tokens and 512 generated tokens with in-flight batching disabled for the efficiency benchmark. Against the best tested TensorRT-LLM precision, QServe improved maximum throughput for Llama 3 8B by 1.2 times on A100 and 1.4 times on L40S. For Qwen 1.5 72B, the reported gains were 2.4 times on A100 and 3.5 times on L40S. The paper's artifact appendix gives an A100 reference result of 2,980.69 generated tokens per second for QServe on Llama 3 8B, versus 2,387.55 for TensorRT-LLM W8A8KV8. Those numbers show that this kernel and quantization co-design worked for the tested models and workload. They do not promise the same multiplier for prefill-heavy traffic, tiny batches, another GPU generation, or another engine version. The[OmniServe repository](https://github.com/mit-han-lab/omniserve)is the implementation check. Its supported models, kernels, conversion scripts, and benchmark settings matter more than the W4A8 label alone. ## When to choose W4A8 Start with the deployment bottleneck. If a small decode batch mostly waits on weight reads, W4 storage is useful. If a large batch is compute-bound, INT8 tensor-core execution is useful. W4A8 is compelling when the engine delivers both without placing floating-point dequantization or partial-sum conversion in the GEMM main loop. Then check KV precision separately. A 4-bit cache can raise batch capacity and attention bandwidth, but it needs an accuracy treatment such as SmoothAttention and a fused kernel that stays memory-bound. Finally, benchmark the actual prompt lengths, output lengths, concurrency, and latency targets. Maximum offline throughput can hide a regression in time to first token or inter-token latency. QServe's contribution is not simply choosing 4 and 8 as smaller numbers. It arranges storage, quantization ranges, weight layout, and GPU instructions so the low-bit representation reaches the hardware fast path. --- This representation contains public Fanout content only. Protected Pro lessons, account data, billing, checkout, and pricing are not included. Browse the public content map: https://fanout.sh/sitemap.md