FP8 vs INT8 vs AWQ vs GPTQ

FP8 vs INT8 vs AWQ vs GPTQ looks like a four-way choice, but the names describe different things.

FP8 and INT8 are eight-bit number formats. AWQ and GPTQ are methods that usually produce four-bit weights while leaving activations at 16 bits.

Choose by the tensors that use low precision, the kernels that execute them, and whether the service needs weight capacity, decode bandwidth, or faster matrix multiplication.

FP8 vs INT8 vs AWQ vs GPTQ is not one axis

FP8 specifies how a value is represented. The FP8 formats paper defines E4M3 and E5M2, which trade mantissa precision against exponent range.

INT8 stores an integer from a fixed set of 256 values. A scale, and sometimes a zero point, maps a range of real values into that set.

AWQ specifies how to prepare and quantize weights while protecting channels that matter to the layer output.

GPTQ specifies an order and an error-correction procedure for quantizing weights with approximate second-order information.

An AWQ model commonly stores INT4 weights. A GPTQ model commonly does the same. Neither name alone states the group size, activation type, kernel, or cache type.

Likewise, an FP8 deployment might quantize weights and activations, weights only, or the KV cache. The label needs a tensor layout such as W8A8 or W4A16 beside it.

First decide what must become smaller

W8A8 means eight-bit weights and eight-bit activations. W4A16 means four-bit weights and 16-bit activations.

That distinction changes where the speed can come from.

A weight-only W4A16 kernel reads fewer weight bytes, then dequantizes or unpacks the weights for multiplication with higher-precision activations.

This helps most when reading weights limits the step. Single-request decode often has that shape because each generated token streams through the model weights.

A W8A8 kernel can perform the matrix multiplication in eight-bit precision with a wider accumulator. It reduces both transfer and arithmetic cost when the hardware has the right instructions.

That can matter more during prefill or large batches, where matrix multiplications have enough tokens to reuse weights and become compute bound.

Fanout's prefill and decode guide explains why one model can have different bottlenecks across the two request phases.

KV-cache precision is a separate decision. The KV cache quantization guide accounts for cache bits, scale metadata, and residual tokens without changing weight precision.

The raw memory arithmetic

Take a 70-billion-parameter model and count only its weights.

At BF16 or FP16, each parameter takes two bytes. The raw weight file is 70 billion times 2 bytes, or 140 GB in decimal units.

At FP8 or INT8, one byte per parameter makes the raw total 70 GB.

At four bits, two parameters fit in one byte. AWQ or GPTQ weights therefore start near 35 GB.

Those are storage floors, not complete GPU requirements. Scales, zero points, packing alignment, unquantized layers, runtime buffers, activations, and KV cache add memory.

The ratios still clarify the first choice. If a model needs to fall below 70 GB, moving from 16 bits to FP8 or INT8 cannot be enough by itself.

Four-bit weights may make it fit, but the runtime also needs a compatible packed kernel. Loading a 35 GB file and expanding every weight to FP16 before each operation can erase much of the latency gain.

Use the raw arithmetic to reject impossible options, then measure actual allocated memory after the engine loads.

FP8 favors range and native low-precision math

E4M3 uses one sign bit, four exponent bits, and three mantissa bits. E5M2 spends one more bit on the exponent and one fewer on the mantissa.

The exponent gives FP8 a nonuniform spacing and a wider useful range than a uniformly spaced integer format with one scale.

This is useful for activations whose magnitude changes across channels or batches. It does not remove the need for scaling, calibration, or higher-precision accumulation.

FP8 fits best when the accelerator and serving engine have native W8A8 kernels for the model's exact shapes.

Current vLLM quantization support shows why hardware must be part of the name. Its FP8, INT8, AWQ, and GPTQ paths support different accelerator generations.

An FP8 model can halve raw weight storage relative to BF16 and speed supported matrix multiplications. It is still twice the raw weight size of an INT4 model.

Choose FP8 when the model already fits at eight bits, the hardware supports the required kernels, and prefill or batched compute is an important part of the target workload.

INT8 depends on scales and outlier handling

For symmetric INT8, a scale maps a real interval around zero to integer values from -127 to 127.

The Transformers quantization guide writes the mapping as q = round(x / scale), with dequantization approximated by scale times q.

One scale for a whole tensor is cheap but can waste most integer levels when a few activation channels contain large outliers.

Per-channel or per-group scales adapt more closely to the data. They also add metadata and can restrict which kernels run efficiently.

SmoothQuant addresses activation outliers by moving quantization difficulty from activations to weights with a mathematically equivalent channel scaling.

That makes W8A8 INT8 practical for more LLM layers. It is different from plain INT8 storage and different again from weight-only INT8.

Choose an INT8 path by its full recipe: which tensors use INT8, the scale granularity, calibration data, accumulator type, skipped layers, and execution kernel.

GPTQ compensates for quantization error

GPTQ is a post-training weight quantization method built around approximate second-order information.

It quantizes a layer in blocks and updates remaining weights to compensate for error introduced by earlier quantization choices.

The calibration inputs estimate how weight errors affect the layer outputs. GPTQ does not retrain the complete model.

Its original paper quantized models down to three or four bits per weight and reported a 175-billion-parameter model processed in about four GPU hours.

Those results describe the paper's models, kernels, and evaluations. A GPTQ file is not automatically faster than every FP8 or INT8 deployment.

GPTQ is useful when four-bit weight capacity is required and the target runtime has a mature kernel for the model's group size, packing order, and GPU.

Calibration data still matters. Test the quantized model on the prompts and tasks the service will receive rather than relying on a generic perplexity set.

AWQ protects activation-salient channels

AWQ starts from the observation that weights connected to large activation channels can have an outsized effect on output error.

It collects activation statistics, identifies salient channels, and searches for per-channel scaling that makes those weights easier to quantize.

The transformation keeps a uniform low-bit weight representation rather than storing a small mixed-precision set of protected weights.

AWQ usually targets W4A16 serving. Like GPTQ, its main capacity gain comes from four-bit weights and its speed depends on packed weight-only kernels.

The method does not mean activations are four bit. It also does not specify whether the runtime uses GEMV, GEMM, Marlin, or another implementation.

Test AWQ when the model must fit at four bits, the serving stack supports its layout, and calibration prompts represent the real activation distribution.

Compare AWQ and GPTQ with the same model revision, bit width, group size, calibration budget, kernel, and workload. Otherwise the method name is not the only variable.

Choose from the bottleneck outward

Start by profiling a BF16 or FP16 baseline at the expected prompt lengths, output lengths, batch sizes, and concurrency.

Use the roofline model guide to separate weight-bandwidth pressure from matrix-compute pressure.

If the weights do not fit at eight bits, test AWQ and GPTQ with a supported four-bit kernel. Compare quality before tuning throughput.

If the model fits at eight bits and prefill or batch compute dominates, test native FP8 and W8A8 INT8 paths on the actual accelerator.

Record allocated weight memory, KV-cache capacity, time to first token, inter-token latency, input tokens per second, output tokens per second, and task quality.

Also record which kernels ran. A configuration label can stay the same while the runtime falls back to a slower implementation for one head dimension or GPU generation.

FP8 versus INT8 is a format and kernel decision. AWQ versus GPTQ is a four-bit weight-calibration decision. The deployment may need one answer from each layer, not one winner across all four names.