Disaggregated prefill and decode with numbers

Disaggregated prefill and decode runs prompt processing and token generation on different GPU pools, then transfers the request's KV cache between them.

That separation can protect inter-token latency and let each phase scale independently. It also duplicates model weights and puts a network transfer directly on the time-to-first-token path.

Start with the KV bytes per prompt and effective transfer bandwidth. Then measure how much phase capacity still meets the latency targets.

Disaggregated prefill and decode separates two bottlenecks

Prefill processes all uncached prompt tokens in parallel. Its large matrix multiplications can make it compute-bound once the prompt batch is large enough.

Decode adds one token per active sequence at each step. It repeatedly reads model weights and a growing KV cache, so low-batch decode is commonly limited by memory bandwidth.

When both phases share a scheduler, a long prefill can lengthen a decode step for requests already streaming. Chunked prefill limits that interruption by splitting the prompt into smaller jobs.

Disaggregation removes the shared execution queue. A prefill worker builds the cache, a connector moves it, and a decode worker starts generation from the received state.

The DistServe paper frames the benefit as independent control of time to first token, or TTFT, and time per output token, or TPOT.

Fanout's prefill versus decode guide covers the two phases before the serving topology is split.

Calculate the KV cache before discussing bandwidth

For standard grouped-query attention, KV bytes per token equal 2 times layers times KV heads times head dimension times bytes per element.

The factor 2 stores one key and one value. Query heads do not enter the cache formula when grouped-query attention uses fewer KV heads.

The Llama 3 report lists 80 layers, 64 attention heads, and 8 KV heads for the 70B model. Its model dimension of 8,192 gives a head dimension of 128.

With BF16 cache elements at 2 bytes, the arithmetic is 2 times 80 times 8 times 128 times 2. The result is 327,680 bytes, or 320 KiB, per token.

An 8,192-token prompt therefore creates 2.5 GiB of KV data for one sequence. A 32,768-token prompt creates 10 GiB.

Those values exclude allocator metadata, transfer headers, hidden states required by some speculative decoders, and any cache blocks already reusable at the destination.

Fanout's KV cache formula guide derives the same shape for memory planning.

Convert KV bytes into a transfer-time floor

A nominal 100 gigabit-per-second link carries at most 12.5 gigabytes per second before protocol and topology losses.

Moving the 2.5 GiB cache from the 8,192-token example has a theoretical floor near 215 milliseconds on that link. Actual time is higher if one path cannot sustain line rate.

At 200 gigabits per second, the same lower bound is about 107 milliseconds. KV quantization to one byte per element would halve the payload, but only if both pools use a compatible cache format.

These figures do not predict a production result. Tensor-parallel shards can transfer through several NICs at once, and implementations can stream one layer's cache while later layers are still computing.

AWS reports single-digit milliseconds for an 8,000-token Llama 3.3 70B transfer on an instance with 3,200 gigabits per second of EFA bandwidth.

The AWS deployment uses GPU-direct RDMA and multiple devices.

Its result does not predict the transfer time on one 100-gigabit path.

Measure effective GPU-to-GPU bandwidth with the same shard layout, page size, dtype, connector, and concurrent traffic planned for serving.

Use a break-even inequality, not a feature flag

Disaggregation improves latency only when the work it removes exceeds the work it adds.

For TTFT, add prefill queue time, prefill execution, KV transfer, routing, and the first decode step. Compare that total with local queueing and local prefill on the decode worker.

For TPOT, compare a decode-only worker with the mixed scheduler at the same arrival rate and prompt distribution. Average TPOT can hide the long pauses users notice at P95 or P99.

The transfer is easiest to justify when prompts are long and uncached, mixed prefill causes tail stalls, and a fast fabric sits between compatible workers.

Short prompts often fail this test. Their prefill finishes quickly, while routing and transfer still impose a fixed cost.

Prefix-cache hits can also reverse the choice. If the decode pool already holds most prompt blocks, recomputing or fetching a small remainder locally may beat transferring a complete cache from another worker.

A router therefore needs prompt length, cache locality, queue state, and topology. Routing every request through prefill is a policy, not an inherent part of disaggregation.

Size the pools from phase goodput

Raw tokens per second is a weak sizing input because a saturated worker may violate TTFT or TPOT targets.

Profile each phase separately and record goodput: the highest request rate that keeps the chosen fraction of requests inside its phase SLO.

DistServe provides a worked example for a 13B model. At its chosen SLOs, one prefill GPU reaches 5.6 requests per second and one decode GPU reaches 10 requests per second.

Two prefill GPUs can supply about 11.2 requests per second, so one decode GPU becomes the 10-request-per-second bottleneck. The balanced topology is two prefill GPUs to one decode GPU.

That three-GPU layout delivers about 3.3 requests per second per GPU. The colocated baseline in the paper reaches about 1.6 under the same goodput definition.

The ratio is workload-specific. Longer outputs increase decode demand. Longer uncached prompts increase prefill demand. Reused prefixes reduce it.

For a fresh trace, estimate prefill demand as arrival rate times average uncached input tokens. Estimate decode demand as arrival rate times average generated tokens.

Divide each demand by its measured per-worker capacity at the target SLO, then round up. Keep headroom for variance rather than sizing both pools to their average utilization ceiling.

Recalculate when prompt length, output length, cache-hit rate, or model parallelism shifts. Fixed one-to-one deployment is not independent scaling.

Goodput and throughput answer different questions

The current vLLM documentation says its disaggregated prefilling feature does not improve throughput.

DistServe reports much higher request goodput under latency constraints. The AWS deployment also reports throughput gains in its tested configuration.

They measure different things. Maximum raw token throughput, request throughput, and SLO-qualified goodput use different denominators and load conditions.

Separating phases can consume more total GPUs because both pools hold model weights. It may lower raw efficiency while serving more requests inside a strict TPOT target.

State the metric before citing an improvement. Include the number and type of GPUs, prompt and output distributions, concurrency, cache hits, and latency percentile.

An apples-to-apples test gives the colocated and disaggregated systems the same hardware budget and applies the same TTFT and TPOT gates.

Compare disaggregation with chunked prefill

Chunked prefill and disaggregation both reduce long interruptions to decode, but they pay in different places.

Chunking keeps one model copy and avoids cross-worker KV transfer. Smaller chunks can protect TPOT, while extra scheduling steps and smaller matrix operations may lengthen TTFT.

Disaggregation lets prefill use a different tensor or pipeline parallel plan and removes execution interference. It pays for another model copy, a connector, and a network handoff.

Fanout's chunked prefill guide explains why the chunk size sets a latency tradeoff rather than acting as a universal optimum.

Start with chunking when one pool has enough capacity and tail latency can be met by tuning the token budget.

Test disaggregation when separate phase capacity, hardware choice, or isolation matters enough to justify the transfer path.

The two techniques can coexist. A prefill pool may chunk very long prompts for fairness or pipeline balance before it streams KV blocks to decode.

Cache compatibility is part of the protocol

The receiver must interpret every transferred block exactly as the sender produced it.

Model revision, layer count, KV-head sharding, head dimension, dtype, attention layout, page size, and positional state must agree.

Tensor-parallel degrees can differ across pools, but then the transfer layer must split or reconstruct shards. A nominally fast fabric may spend its time on many small or strided copies.

The decoder also needs enough free blocks before accepting a request. If prefill pushes faster than decode consumes, transferred caches can exhaust decode memory and turn isolation into backpressure.

DistServe uses a pull design so decode workers fetch caches as capacity becomes available. Other systems use shared cache tiers, direct writes, or explicit admission buffers.

Failure handling needs the same precision. Decide who retries after a partial transfer, how stale cache blocks are identified, and whether a request can fall back to local prefill.

Network transfer now sits on the serving critical path. Monitor queue time, bytes, bandwidth, retries, and destination allocation failures beside TTFT and TPOT.

Decide from a trace and a topology

First, calculate KV bytes for the actual model, cache dtype, and uncached prompt length.

Second, benchmark effective transfer time across the planned GPU and NIC topology. Do not substitute the fabric's marketing rate.

Third, profile prefill-only and decode-only goodput under separate TTFT and TPOT targets. Use the trace's prompt, output, and cache-hit distributions.

Fourth, compare against a well-tuned colocated baseline with chunked prefill. Hold the hardware budget and SLO gates constant.

Fifth, choose a routing threshold from measured break-even points. Keep short or cache-hot requests local when transfer would add time.

Finally, test overload and failure. A balanced average ratio can still produce a prefill queue, decode memory exhaustion, or a burst of simultaneous KV transfers.

Disaggregated prefill and decode is worthwhile when isolation and independent capacity recover more SLO-qualified work than the extra model copies and cache movement consume.