DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving
Prefill is compute-bound and decoding is bandwidth-bound, so sharing a GPU makes each wait on the other. Follow DistServe's split into prefill and decode pools, the per-GPU goodput arithmetic, the KV-cache transfer it adds, and what its evaluation does and does not show.
Reading focus: Why time to first token and time per output token pull a shared GPU in opposite directions, and how goodput scores both. How separate prefill and decode pools turn capacity planning into a ratio between two phase rates, with a KV-cache handoff between them. What the placement search, the transfer measurements, and the vLLM++ ablation establish, and where synthetic arrivals and author-set deadlines bound the claim.
OSDI 2024. Yinmin Zhong, Shengyu Liu, Junda Chen, Jianbo Hu, Yibo Zhu, Xuanzhe Liu, Xin Jin, and Hao Zhang. 16 min read, medium difficulty.