AI inference engineering, explained with numbers
AI inference engineering is the work of running trained models in production at a cost and latency the product can survive. Most guides define it that way, then list the same six techniques as if each were a separate invention.
They form one family. Nearly every technique in AI inference engineering attacks the same number: a modern GPU can do far more arithmetic than its memory can feed. Work through that ratio once and the field stops looking like a list.
The gap that creates the discipline
An H100 SXM delivers close to 1,000 teraFLOPS of dense BF16 matrix throughput against 3.35 TB per second of memory bandwidth.
To keep the tensor cores busy, every byte fetched from memory has to support roughly 295 floating point operations.
Generating one token does nowhere near that. A forward pass costs about two floating point operations per parameter while reading two bytes per parameter at BF16: one operation per byte moved.
One versus 295. During single-stream generation the arithmetic units spend most of each step waiting on memory, and the whole discipline exists to close that gap.
This framing is the roofline model applied to transformers. Fanout's roofline guide covers where the crossover sits and how to compute it for your own kernels.
Prefill and decode sit on opposite sides of the ratio
Processing the prompt, called prefill, pushes thousands of tokens through one read of the weights. Arithmetic per byte is high, the phase is compute-bound, and its user-facing metric is time to first token.
Generating the response, called decode, produces one token per sequence per step. Every step re-reads the weights, so the phase is bandwidth-bound. Its metric is inter-token latency, or its inverse, tokens per second.
The ceiling is easy to compute. Llama-3.1-70B stored at 8 bits is 71 GB of weights, and 3,350 GB per second divided by 71 GB caps single-sequence decode near 47 tokens per second on one H100.
Faster tensor cores would not move that number at all.
Every optimization below is easier to place once you know which phase it serves. The prefill and decode guide walks through the split in detail.
Batching reuses the bytes you already paid for
Weight bytes are read once per step no matter how many sequences share the pass. Serve 32 sequences together and the same 71 GB of traffic yields 32 tokens instead of one, multiplying arithmetic per byte by 32.
Batching is therefore not a throughput trick bolted onto inference. It is the direct fix for the one-versus-295 gap, which is why every serving engine is built around it.
The scheduling problem is that requests arrive and finish at different times. Continuous batching admits and retires sequences at every step instead of waiting for the slowest member of a batch.
The Orca paper introduced the approach and reported a large throughput gain over its FasterTransformer baseline.
Fanout's continuous batching guide covers how the schedulers differ.
The KV cache decides how far batching scales
Each active sequence keeps its attention history in GPU memory as a KV cache, and that footprint grows with context length and batch size. Batch capacity usually runs out of cache memory long before it runs out of compute.
The KV cache formula converts your model and context length into gigabytes per sequence.
Two techniques stretch that budget. PagedAttention allocates the cache in small blocks instead of contiguous slabs, cutting fragmentation waste.
The vLLM paper reported 2 to 4 times the throughput of prior systems at the same latency. Fanout's PagedAttention post explains the block table.
Prefix caching reuses cache blocks across requests that share an identical prompt prefix, such as a system prompt or an ongoing conversation. The shared portion skips prefill entirely.
Character.AI credited inter-turn caching as one reason it served around 20,000 queries per second at 33 times less cost than when it launched in 2022.
Quantization shrinks every byte the ceiling counts
In a bandwidth-bound regime, speedup is proportional to bytes removed. Quantize the 71 GB model to 4 bits and it becomes 35 GB, lifting the single-sequence ceiling from about 47 tokens per second to about 95 on the same GPU.
Accuracy is the obvious worry, and it is more settled than its reputation. A 500,000-evaluation study across the Llama-3.1 family found FP8 effectively lossless.
It also found 4-bit weights more competitive than expected. Task-specific evals remain your responsibility.
Which scheme to pick depends on batch size and hardware generation, and the answer flips between regimes. The W8A8 vs W4A16 comparison works through that decision.
Speculative decoding buys more tokens per weight read
If reading all the weights is the expensive part, get more than one token out of each read. A small draft model proposes several tokens cheaply, and the large model verifies the whole run in a single pass, keeping the accepted prefix.
Leviathan et al. reported 2 to 3 times faster generation on T5-XXL with unchanged output distribution.
The gain concentrates at small batch sizes, where idle arithmetic capacity makes verification nearly free. Under saturated batching that slack disappears.
The speculative decoding guide covers draft choices and acceptance math.
Parallelism and disaggregation reshape the hardware
When one GPU's bandwidth is the wall, tensor parallelism splits each layer across several GPUs so their memory channels work in aggregate. Eight H100s present about 26.8 TB per second.
The price is per-layer communication over NVLink. The trade-offs against pipeline and expert variants are in Fanout's parallelism comparison.
Colocating the two phases creates a subtler problem. A long prefill occupies the GPU for whole seconds, stalling every decoding sequence that shares it and spiking inter-token latency.
Disaggregation runs prefill and decode on separate GPU pools and ships the KV cache between them.
DistServe reported serving 7.4 times more requests, or meeting 12.6 times tighter latency objectives, once each phase could be provisioned and scaled on its own.
The disaggregation post examines when the cache transfer is worth it.
When AI inference engineering pays for itself
None of this is free to operate, so early products should buy inference from an API and move on.
The switch point is arithmetic, not fashion. It arrives when the API bill, the latency floor, or the provider's uptime becomes the binding constraint on your product.
Open weights made the switch realistic. Hugging Face hosts over two million model repositories.
Since DeepSeek-V3, the strongest open models have stayed within months of the closed frontier.
Character.AI's figure from the post above, serving traffic at 13.5 times less than leading commercial APIs would charge, shows the ceiling on what a dedicated stack can recover.
Before committing, put numbers on your own workload with the API cost estimation guide. A few hours of arithmetic is cheaper than a cluster.
The map, in short
- Batching and speculative decoding raise the work extracted from every byte of weight traffic.
- Quantization and KV cache management shrink the bytes themselves.
- Tensor parallelism buys aggregate bandwidth; disaggregation stops prefill from stealing it mid-decode.
- Prefix caching deletes repeated work before either phase begins.
Every one of these is a response to the same ratio from the first section.
To go deeper in sequence, the inference engineer roadmap orders the topics, and Fanout's inference engineering course works through them with the arithmetic attached.