The LLM inference engineer roadmap

An LLM inference engineer makes trained models answer requests within a latency, throughput, memory, reliability, and cost budget.

The job sits between model architecture, GPU programming, distributed systems, and production operations.

You do not need to master those fields independently before starting. You need to follow one request from tokenization to the last generated token and explain where time and memory went.

Begin with the shape of one request

An autoregressive transformer generates one token, appends it to the sequence, and repeats.

The first pass over the prompt is prefill. It processes many prompt tokens in parallel and creates the key-value cache.

Decode follows. Each step handles a small amount of new work and reads the growing cache.

These phases have different bottlenecks. Prefill can use large matrix operations efficiently. Decode often moves more memory relative to the arithmetic performed.

Read prefill vs decode before treating "tokens per second" as one number.

Build a tiny benchmark that records prompt length, output length, time to first token, inter-token latency, total latency, and memory use.

Learn enough transformer math to count work

Know the dimensions of embeddings, attention projections, feed-forward layers, heads, and key-value heads.

For every matrix multiplication, identify the input shape, weight shape, output shape, and numeric format.

Calculate parameter memory. Estimate activation memory. Derive KV cache bytes per token from layers, key-value heads, head dimension, bytes per element, and keys plus values.

Then check the estimate with Fanout's KV cache calculator.

Use the formula to catch an impossible configuration before loading a model.

Grouped-query and multi-query attention reduce KV cache size by sharing key-value heads. They change memory and bandwidth without shrinking every model component.

Understand the GPU as a memory hierarchy

GPU performance is not "more cores means faster."

Learn warps, blocks, occupancy, registers, shared memory, device memory, memory coalescing, and synchronization.

The CUDA C++ Best Practices Guide connects these ideas to measurement.

Use a profiler before writing a custom kernel. Determine whether the operation is limited by compute, memory bandwidth, launch overhead, synchronization, or poor shapes.

Implement one simple reduction or matrix-related kernel. Compare it with an optimized library and explain the gap.

Most production work uses mature kernels. Writing one yourself teaches why layout, fusion, and launch count matter.

Treat the KV cache as a memory allocator problem

The KV cache grows with active sequence length and number of requests.

Naive contiguous allocation wastes memory when sequence lengths vary or reservations exceed actual output.

PagedAttention applies ideas from virtual memory to store cache blocks non-contiguously.

That lets a serving system share physical capacity across sequences with less fragmentation.

Read the PagedAttention explanation, then inspect how a serving engine maps logical blocks to physical storage.

Ask what happens during beam search, prefix sharing, eviction, preemption, and very long contexts. An allocator design is defined by its failure paths.

Scheduling decides who gets the GPU

Static batching waits for a group of requests to finish together. Short requests can sit behind long ones.

Continuous batching admits new work as slots become available. It increases utilization but makes scheduling and memory management more dynamic.

Orca described iteration-level scheduling for generative models.

Study continuous batching and build a simulator with requests of different prompt and output lengths.

Compare first-come-first-served, shortest-job-first approximations, and priority classes.

Track queue time, time to first token, inter-token latency, throughput, and starvation. An average can improve while an important user class gets worse.

Parallelism is a communication decision

Tensor parallelism splits operations within a layer. Pipeline parallelism splits layers into stages. Data parallelism replicates the model.

Each form moves or duplicates different data.

Tensor parallelism adds collective communication to many layers. Pipeline parallelism creates stage boundaries and bubbles. Replication costs model memory but isolates requests.

Sequence and context parallel methods address long inputs with their own communication patterns.

Choose parallelism from the model size, hardware topology, traffic, and latency goal. Do not begin from the number of GPUs you happen to own.

Draw the topology. Mark which links are NVLink, PCIe, or network. Estimate bytes moved per generated token.

Quantization trades several resources at once

Quantization can reduce weight memory and bandwidth. It may also change kernel availability, accuracy, calibration work, and operational complexity.

Weight-only quantization helps when weight reads dominate. KV cache quantization targets a different memory pool.

Lower precision does not guarantee lower latency. A format with poor kernel support may lose to a larger format with a tuned path.

Evaluate model quality on representative tasks. Measure prefill and decode separately. Record memory, latency distribution, throughput, and power if it matters.

Keep the unquantized baseline and the exact calibration procedure. "INT4" is not a complete experiment description.

Serving engines are systems to inspect

Start with one engine such as vLLM, SGLang, or TensorRT-LLM.

Trace model loading, memory allocation, request admission, tokenization, scheduler decisions, kernel execution, streaming, cancellation, and metrics.

Do not compare engines from default commands alone. Align model revision, precision, context limits, batch policy, sampling, and hardware.

A useful comparison includes low-load latency and saturated throughput. It also includes failed requests, timeouts, and behavior near memory limits.

The serving layer needs backpressure. Without it, a queue can turn overload into minute-long latency and expensive abandoned work.

Speculative decoding changes the decode loop

Speculative decoding proposes several tokens using a cheaper method, then verifies them with the target model.

When enough proposals are accepted, one target-model pass advances by more than one token.

The method can reduce inter-token latency without changing the target distribution when implemented with the correct acceptance rule.

It is not free throughput. A weak draft model, low acceptance, high traffic, or extra memory pressure can erase the gain.

Use the speculative decoding guide after you can explain the ordinary decode path.

Production inference starts with workload traces

A benchmark without a workload shape answers a narrow question.

Collect distributions for prompt length, output length, arrival rate, concurrency, model choice, sampling, tool use, and cancellation.

Replay a sanitized trace or synthesize one that preserves those distributions.

Use open-loop load when you want to control arrival rate. Use closed-loop load when you want to model a fixed set of clients waiting for responses.

Measure percentiles and goodput. Goodput counts requests that meet the service objective, rather than rewarding work users would consider too slow.

Fanout's latency numbers lab helps build intuition for the time scales beneath a request.

Reliability belongs in the roadmap

Test cancellation, worker loss, GPU out-of-memory errors, tokenizer failure, malformed input, deploy replacement, and a dependency slowdown.

Decide when a request can retry. Streaming output makes replay visible and may make an automatic retry unsafe.

Expose queue depth, active sequences, cache occupancy, tokens processed, scheduler delay, GPU utilization, errors, and request latency.

Add admission limits before the first traffic spike. An inference server should reject work clearly before it accepts more than it can finish.

A project sequence that builds evidence

Project one: benchmark one open model on one GPU. Separate prefill and decode and validate memory estimates.

Project two: add concurrent requests with varied lengths. Implement a scheduler simulator and compare it with observed engine behavior.

Project three: test quantization, prefix caching, or speculative decoding with quality and performance baselines.

Project four: serve across multiple GPUs. Document topology, parallelism, communication, and failure handling.

Project five: replay a production-shaped trace with service objectives, dashboards, backpressure, and a cost model.

Publish the methodology and raw configuration, not only the winning number.

The Fanout inference engineering path turns these topics into a broader course sequence. Start by profiling one request until every millisecond has an owner.