What is chunked prefill?

What is chunked prefill? It is a scheduling method that splits a long prompt into smaller pieces and processes those pieces across several model iterations.

The model still reads every prompt token before generating the response. Chunking changes when prompt work runs, so existing decode streams do not sit behind one uninterrupted prefill.

The setting is a tradeoff, not a free speedup. Smaller chunks protect streaming latency. Larger chunks can finish a new prompt sooner and use the GPU more efficiently.

What is chunked prefill solving?

An LLM request starts with prefill. The model processes known prompt tokens in parallel, writes their keys and values to the KV cache, and produces the first output token.

Decode follows one token at a time. Each active sequence contributes a small amount of new work and repeatedly reads model weights and cached state.

The SARATHI paper describes prefill as compute-saturating work and decode as low-utilization work when only one token per request is processed.

Without chunking, a long prefill can create one large scheduler iteration. Decode requests that were streaming smoothly may wait until that iteration completes.

Users experience the pause as worse inter-token latency, or ITL. The new request is trying to improve its time to first token, or TTFT, while existing requests need frequent decode steps.

The problem is interference between two different workloads on the same device. Fanout's prefill vs decode guide develops that distinction from the request trace.

Chunking creates more scheduling boundaries. The scheduler can advance decodes between pieces of the long prompt instead of letting that prompt own one long iteration.

A 10,000-token prompt worked example

Assume the scheduler can process at most 2,048 tokens in one iteration. This is the token budget, not the model's maximum context length.

Also assume 64 requests are already decoding. Each contributes one token to the next iteration, so decodes consume 64 slots first.

The remaining prefill budget is 2,048 - 64 = 1,984 tokens.

A new 10,000-token prompt therefore needs ceiling(10,000 / 1,984) = 6 chunks. Five chunks process 9,920 tokens, and the sixth processes the remaining 80.

During each mixed iteration, the 64 active requests can advance one decode token before the scheduler spends the rest of its budget on prefill.

Without chunking, the 10,000-token prefill could run as one long unit and delay those decode streams. With chunking, its work is spread across six shorter scheduling opportunities.

The new request does not generate output after its first chunk. It must complete all six before decode begins, so its TTFT includes several iterations.

The arithmetic matters because chunking does not reduce 10,000 prompt tokens to 1,984. It changes the maximum amount of prompt work admitted at once.

The token budget moves TTFT and ITL

Use the same 10,000-token prompt and 64 active decodes with a smaller budget of 1,024 tokens.

After decodes take 64 slots, 960 remain for prefill. The prompt now needs ceiling(10,000 / 960) = 11 chunks.

Each iteration contains less prefill work, which can shorten the delay imposed on streaming requests. The long prompt needs more rounds before its first token.

Now raise the budget to 8,192. After 64 decode slots, 8,128 remain. The prompt needs only two chunks.

The larger budget can improve TTFT for the waiting prompt and may raise throughput. Each mixed iteration is heavier, so decode streams can see longer gaps.

Current vLLM optimization guidance states the same direction: smaller token budgets favor ITL, while larger budgets favor TTFT and throughput.

A blog post cannot supply the right budget. It depends on the model, GPU, kernels, prompt distribution, active sequence count, and latency objective.

Treat 1,024, 2,048, and 8,192 as experiment points, not universal presets.

Decode-first scheduling makes chunking useful

Splitting a prompt is only half the mechanism. The scheduler must decide what fills each iteration.

In vLLM V1, chunked prefill normally works with decode-first scheduling. Pending decode tokens enter the batch, then prefill work uses the remaining token budget.

If a waiting prefill does not fit, the scheduler admits a partial prefill and leaves the rest for a later iteration.

This policy protects active streams because they are considered before new prompt work. It also creates mixed batches containing decode tokens and prefill tokens.

The SARATHI paper calls its related policy decode-maximal batching. Decode requests piggyback on a prefill chunk that supplies enough compute work to use the GPU more fully.

The later Sarathi-Serve paper builds stall-free schedules around near-equal prefill chunks and reports capacity gains under explicit tail-latency constraints.

Those reported gains belong to the evaluated models, hardware, and workloads. The mechanism is general, but the multiplier is not.

Chunked prefill also complements continuous batching.

Continuous batching changes which requests participate between iterations. Chunking controls how much of a prompt can participate in one iteration.

Chunking preserves the full prompt

Chunked prefill does not summarize, truncate, or independently encode each chunk.

The first chunk runs through every transformer layer and writes KV entries for its tokens. The next chunk attends to the prior cached tokens and appends its own entries.

After the final chunk, the KV cache represents the full prompt. Decode proceeds from the same logical context the model would have after an unchunked prefill, subject to the engine's numerical behavior.

This works because prompt tokens are already known. Their causal attention states can be computed in order across several forward passes.

Decode cannot generate an arbitrary block of future tokens the same way. Token n + 1 depends on the sampled result at token n.

The SARATHI analysis notes a cost: later chunks reread KV pairs from earlier chunks during attention.

Very small chunks can therefore add substantial attention and launch overhead. The paper found that chunk size and hardware tile boundaries affected prefill efficiency in its experiments.

The context is preserved, but the cost of building it can rise.

Memory benefits need careful wording

Chunking can reduce the amount of temporary activation work admitted in one iteration. Some runtimes use that bound to size buffers more predictably.

It does not remove the final KV-cache requirement for the full prompt. Once all 10,000 tokens are prefetched, their retained key and value states still occupy cache memory.

NVIDIA's TensorRT-LLM chunked prefill guide describes dynamic chunk sizing and bounded activation buffers.

That is different from claiming context memory becomes independent of sequence length. KV storage still grows with cached tokens unless another mechanism changes its representation or placement.

The PagedAttention guide explains how block allocation reduces waste around that cache. Chunked prefill changes the schedule; PagedAttention changes cache allocation.

When memory pressure triggers preemption or recomputation, lowering the token budget or concurrent sequence limit may help. It may also reduce throughput.

Measure peak temporary memory and steady KV-cache occupancy separately.

When chunked prefill helps most

Chunking is useful when prompt lengths vary and long prefills disturb active streams.

Interactive chat beside document ingestion is a common example. Short conversations need steady decode cadence while an occasional long prompt needs substantial prefill compute.

It can also help pipeline-parallel inference. More uniform chunks reduce iteration imbalance, which can reduce bubbles between stages.

The original SARATHI evaluation reported large decode-throughput gains and smaller end-to-end throughput gains for its tested models. Sarathi-Serve later evaluated serving capacity under latency limits.

Reported multipliers matter less than the operating condition. Chunking is most useful when mixed prefill and decode traffic leaves hardware idle or causes latency stalls.

It may help less in an offline prefill-only workload, a quiet server with no overlapping decodes, or a system where kernels already dominate at the chosen chunk size.

If no single budget satisfies both TTFT and ITL targets, separate prefill and decode workers may be the next design. That adds KV transfer, routing, capacity planning, and failure modes.

Use disaggregation after shared scheduling has a measured limit, not as a default reaction to one long prompt.

Tune it from traces, not averages

Record prompt tokens, active decodes, scheduled prefill tokens, scheduled decode tokens, iteration duration, and queue time for each step.

Then graph long-prefill volume against TTFT and ITL percentiles. Look for larger prefill chunks producing visible stalls in requests that are already streaming.

Run a sweep of token budgets under the same arrival trace. Keep model revision, precision, cache size, maximum sequences, and sampling settings constant.

Measure at least:

• TTFT percentiles by prompt-length bucket.
• ITL percentiles for streams overlapping a long prefill.
• Completed input and output tokens per second.
• Request goodput under the latency objective.
• GPU utilization and iteration duration.
• KV-cache preemptions and recomputations.

Pick the smallest budget that meets throughput and TTFT requirements without breaking the ITL target. If several values pass, prefer the one with more headroom under the expected prompt mix.

Re-run the sweep after changing the model, quantization, attention backend, GPU, or maximum concurrency. Those changes alter the useful chunk size.

Chunked prefill is a scheduler boundary. Its value comes from making a long prompt share the machine predictably, not from making the prompt disappear.