LLM serving latency decomposition

Separates a request's waiting, prompt-processing, and iterative decode contributions so bottlenecks are not hidden in one average.

Request latency is approximately queueing time plus prefill time plus output token count minus one times time per output token.

Total response time is waiting plus prompt work plus the cadence cost of every generated token after the first.

Addition composes sequential critical-path phases, while multiplication expands the repeated decode-token phase.

Continuous batching, preemption, variable TPOT, speculative decoding, and overlap can make real request traces depart from this simple additive model.