Prefill vs decode in LLM inference
One LLM request creates two different workloads on the same model.
First, the engine processes the prompt and builds its KV cache. Then it generates output one token at a time while reading that cache.
Calling both phases “inference” is correct but operationally incomplete. They stress the hardware differently, expose different latency metrics, and compete when a scheduler mixes them carelessly.
This guide uses the DistServe paper and current vLLM scheduling guidance as primary references.
Prefill turns the prompt into model state
During prefill, the model processes the input tokens and produces the first output token. It also writes the prompt’s keys and values into the KV cache.
Prompt tokens are already known. The engine can process many of them together through the model’s matrix operations.
A longer prompt therefore creates more prefill work before the user sees the first generated token.
The user-facing metric is time to first token, or TTFT. It includes more than GPU execution: queueing, tokenization, network overhead, and scheduling can all contribute.
When diagnosing TTFT, separate those components before blaming the model kernel.
Decode advances one dependent step at a time
After prefill, the model generates a token, appends its new keys and values, then uses that token as input for the next step.
The next token depends on the previous result. That dependency prevents a single sequence from generating its whole answer in parallel.
The engine recovers parallelism by decoding many sequences together. Each active sequence usually contributes one new token to an iteration.
The user feels decode performance as the delay between streamed tokens. Systems report it as time per output token or inter-token latency.
Output length multiplies this phase. A fast first token can still lead to a slow completion when hundreds of sequential decoding steps follow.
The two phases have different arithmetic shapes
Prefill sends many prompt tokens through the model at once. Large matrix operations can use the accelerator’s compute capacity efficiently.
Decode handles a small number of new tokens per active sequence and repeatedly reads model weights and the growing cache.
For many common serving configurations, prefill trends toward compute-bound work while decode trends toward memory-bandwidth-bound work.
That is a workload tendency, not a law. Model architecture, batch size, quantization, parallelism, kernels, and hardware can move the bottleneck.
Measure the actual engine. A slogan about compute and bandwidth is only a starting hypothesis.
Fanout’s latency numbers Lab is useful for rebuilding intuition about the memory and communication gaps underneath these phases.
TTFT and inter-token latency can move in opposite directions
A scheduler can favor prefill work to admit prompts quickly. That may improve TTFT while making existing decode streams pause behind large prompts.
It can instead prioritize decode tokens. Existing streams stay smooth, but new requests wait longer for their first token.
The tradeoff becomes visible under mixed traffic. A short chat turn, a long document prompt, and a long generated report place different pressure on the same queue.
This is why average end-to-end latency is not enough.
- TTFT shows the wait before generation starts.
- Inter-token latency shows the rhythm of streaming generation.
- End-to-end latency includes both phases and the requested output length.
- Throughput shows how much total work the server completes.
A serving change is not “faster” until you say which metric improved and what happened to the others.
A long prefill can stall many decodes
Imagine 40 conversations already decoding smoothly. A new request arrives with a prompt several times longer than the rest.
If the engine runs that prefill as one large unit beside the decode batch, existing streams can experience a latency spike.
The issue is interference. A large compute-heavy prompt enters a loop whose other participants need small, frequent decode steps.
The DistServe paper treats TTFT and time per output token as separate service objectives because colocating the phases couples their resources and latency.
The problem is visible on a single shared server. Plot inter-token latency against prefill token volume while both phases use the same GPU.
Chunked prefill creates scheduling boundaries
Chunked prefill divides a long prompt into smaller token chunks rather than processing the entire prompt in one uninterrupted pass.
The scheduler can place decode tokens between those chunks. This limits how long one prompt can monopolize an iteration.
The Sarathi-Serve paper studies this idea as stall-free scheduling that piggybacks decode work with chunked prefills.
Current vLLM V1 scheduling prioritizes pending decode work, then uses the remaining token budget for prefills. A prefill that does not fit is split into another chunk.
The idea is simple: turn one large scheduling object into several smaller ones.
The tradeoff lives in the chunk budget.
A smaller token budget can protect inter-token latency because less prefill work joins each iteration. It may increase TTFT because the prompt needs more scheduling rounds.
A larger budget can finish prefills sooner but create heavier iterations for active decodes.
Chunking is not the same as continuous batching
Continuous batching changes which requests can join or leave between model iterations.
Chunked prefill changes the size of prompt work admitted to an iteration.
An engine can support iteration-level scheduling without splitting a long prefill. It can also chunk prefills while using a particular priority policy for decode.
The two mechanisms work well together because both make scheduling more granular.
The companion guide to continuous batching follows the request lifecycle through that loop.
Disaggregation gives each phase its own resources
Chunking reduces interference on shared hardware. Disaggregated serving goes further by sending prefill and decode to different workers.
A prefill worker processes the prompt and creates the KV cache. That state must then move to a decode worker that generates the remaining tokens.
Separate pools can use different parallelism plans and scale independently around their own latency targets.
The price is coordination. The system must transfer cache state, route the request correctly, provision another copy of model weights, and account for network bandwidth and failures.
DistServe reports strong goodput gains under its evaluated clusters and service-level objectives. Those numbers are evidence for the design, not a promise for every deployment.
Disaggregation becomes attractive when measured phase interference and independent scaling needs justify its operational cost.
Prefix caching mainly reduces repeated prefill work
If many requests share an identical prefix, the server may reuse the prefix’s KV blocks instead of recomputing them.
That can reduce TTFT and prefill compute for repeated system prompts, templates, or document prefixes.
It does not make novel suffix tokens free. The unmatched prompt portion still needs prefill, and generated tokens still need decode.
Cache-hit rate also depends on exact tokenized prefixes and validity boundaries. Small prompt changes can prevent reuse.
The PagedAttention guide explains how block-based cache management supports allocation and sharing without changing the logical attention result.
Capacity planning must count both token streams
Requests per second hides the shape of the work. Two requests can have the same arrival rate and radically different prompt and output token volumes.
Track at least:
- Input tokens arriving per second
- Output tokens generated per second
- Prompt length percentiles
- Output length percentiles
- Concurrent decoding sequences
- KV-cache utilization
- TTFT and inter-token latency percentiles
Prefill capacity follows the incoming prompt workload. Decode capacity follows active sequences and the output tokens they still need.
The two phases meet in the KV cache. Prefill creates it, decode extends and reads it, and the scheduler decides how many live sequences may keep it resident.
Fanout’s KV-cache calculator makes the memory side explicit before you add queueing and token budgets.
A practical diagnosis order
Start with a trace for one request. Mark queue entry, prefill start, first token, each decode step, and completion.
Next, group traffic by prompt and output length. Do not let one average conceal a small number of very long requests.
Then compare quiet and loaded runs. If isolated kernels are fast but TTFT rises under load, queueing or admission may dominate.
If streamed tokens become uneven when large prompts arrive, inspect mixed prefill-decode batches and the chunked-prefill budget.
If the cache repeatedly fills, inspect preemption and recomputation before tuning kernels.
Only after the phase boundary is visible should you consider separate prefill and decode pools.
The useful mental model
Prefill is not “the first decode step,” and decode is not “prefill with one token.” They share weights but create different serving problems.
Prefill asks how quickly the system can turn a known prompt into reusable state.
Decode asks how steadily it can advance many dependent token streams while preserving that state.
The scheduler mediates between them. Its priorities show which latency promise the system protects when traffic becomes difficult.
That is why prefill versus decode belongs near the beginning of Fanout’s inference engineering roadmap, before kernel tuning or fleet-scale architecture.