TTFT, TPOT, ITL, E2E & Throughput

Watch a served model answer and the reply has a shape: a wait before the first word, then a steady drip of words until the last one. curl -N from HTTP serving printed that shape in a terminal, and the lesson on prefill, decode, and KV memory explained the machinery behind it, one prefill pass and then one decode step per token. The cached-generation lab closed with an envelope in memory.md saying what your box can hold, and promised this module's measurement discipline as the next step. The discipline starts with vocabulary: the names the industry attaches to that shape, and what exactly each one measures.

One request on a timeline

Put a clock on the shape. Four kinds of instant matter: the moment the request is sent, the moment the first token arrives, the moments each later token arrives, and the moment the last one does. Every metric in this module is a difference between two of those stamps, which is worth holding onto as the acronyms accumulate.

To have real stamps to subtract, this lesson uses a capture from the course CPU reference box: 200 sequential greedy requests of the canonical prompt through the cached loop from the KV-cache implementation, 16 new tokens each. A wall-clock stamp was recorded as each token reached the measuring code. Absolute times are CPU-box times, per the pattern from prefill, decode, and KV memory; the relationships are what transfer to a GPU. The first request paid one-time costs and is excluded from every latency summary below; what it paid, and the discipline of excluding it, is covered in warm-up and steady-state measurement.

The four latency names

The lesson on prefill measurement measured the wait before the first word, identified it as prefill running, and left a promise: the industry's name for that wait. The name is time to first token, TTFT: the clock starts when the request leaves the client and stops when the first token arrives back.

The capture's median TTFT is 0.277 s, against the 0.26 s prefill step the lesson on prefill, decode, and KV memory measured on the same box and prompt. The difference is everything else a user waits through besides prefill: tokenizing the prompt, selecting the first token, delivering it to the measuring code. On a real serving stack one more component joins them, and it can dwarf the rest: time spent waiting in line before prefill even starts. In this capture the line was always empty; TTFT was prefill plus small change.

The gaps after the first token have their own name: inter-token latency, ITL. Each gap is one decode step as the client experiences it, and the flat measured line from decode measurement is why one name covers all of them: the second gap and the fifteenth cost the same because the cache absorbed the past. Across the capture's kept runs there are 2,985 gaps with a median of 0.0783 s, the same 0.08 the flat line showed from inside the process, now stamped from outside it. A single 16-token request contributes fifteen gaps, so ITL arrives as a distribution; the worst gap in the kept runs is 0.2356 s, three times the median, which a user would feel as a stutter. This lesson stops at medians and one worst case; what the far end of the distribution costs a product is covered in percentiles and variance.

Time per output token, TPOT, is the averaged version of the same gaps: TPOT = (E2E − TTFT) ÷ (output tokens − 1), one number per request. Since a request's gaps sum to exactly its E2E minus its TTFT, TPOT is the mean of that request's ITLs. For run 10 that is (1.4345 − 0.2551) ÷ 15, or 0.0786 s per token. The compression is both the point and the cost: run 63 contains the capture's worst gap, that 0.2356, yet averages out to a TPOT of 0.0984 s, nothing remarkable. TPOT answers how fast a request decoded on average; the ITLs answer whether the stream stuttered. Keep both words because they answer different questions.

End-to-end latency, E2E, is the whole request, sent to last token, and it closes the arithmetic: E2E is TTFT plus the sum of the ITLs. The capture obeys the identity on every one of the 200 runs to within 0.3 ms, the rounding of the stamps themselves. The median E2E is 1.461 s at 16 output tokens. E2E is the number a non-streaming caller feels, because without streaming nothing arrives until everything has; and at a fixed rhythm it is roughly TTFT plus output length times the typical gap, which makes output length the strongest lever on it.

MetricThe subtractionPhase that owns itWhat moves it
TTFTrequest sent → first tokenprefill, plus the line in front of itprompt length, queue depth
ITLone token → the nextdecodeweight stream, cache growth, contention
TPOT(E2E − TTFT) ÷ (tokens − 1)decode, averaged per requestsame as ITL
E2Erequest sent → last tokenboth phasesoutput length above all

Throughput and its denominators

The lesson on memory and throughput arithmetic filed latency and throughput in separate unit families, time per event against events per second. At the serving level one question remains: which event to count. The answer changes the number, and these rates divide the whole session's clock, first request included, because total work over total time is what a rate is.

Count output tokens: the capture produced 3,200 of them over 307.6 s of wall clock, 10.4 tok/s. Count every token the server processed, prompt plus output, 22 per request: 14.3 tok/s. Count requests: 0.65 req/s. Three true numbers from one run, the first two more than a third apart, so a throughput claim without its denominator is not yet a claim. Serving benchmark reports print all three lines for exactly this reason.

For a single stream, the reciprocity from memory and throughput arithmetic holds, with fine print about which stretch of clock you invert. The median gap inverted says decode runs at 1 ÷ 0.0783, about 12.8 tokens per second. A whole request divided out, 16 tokens over 1.461 seconds, gives 11.0, the TTFT diluting the rate. The whole session gives the 10.4 above. All three are honest; each divides a different stretch of the same clock, and a report should say which.

Under batching the reciprocity ends, exactly as the misconception box from the lesson on memory and throughput arithmetic warned; the serving vocabulary just gives the warning names. Sharing one weight stream across requests multiplies the system's output tokens per second while each stream's own ITL holds or worsens. Throughput is the system's number and ITL is each user's; connecting them takes one more number, how many streams share the machine.

What the tools mean by these words

The definitions above are the course's, and they match the benchmark client built into vLLM. As of August 2026 its serve benchmark computes TPOT by the same subtraction, pools every request's raw gaps into the ITL distribution, and reports mean, median, and percentile lines per metric under its own column names. One wrinkle: its ITL stamps gaps between streamed chunks, which equals per-token gaps only when the server streams one token per chunk, the usual case.

GuideLLM, a dedicated benchmarking tool from the same ecosystem, uses the same words differently: its ITL is a per-request average of the gaps, close to what vLLM calls TPOT, and its time per output token folds the first token into the average rather than excluding it. The two dictionaries disagree on the same data: average run 10's gaps and you get 0.0786 s per token; fold the first token in, 1.4345 over 16, and you get 0.0897 s, fourteen percent apart by definition alone. So the module's first habit: a benchmark number means nothing until you know which subtraction produced it. When a report crosses your desk, find the tool's definitions before comparing its columns with anyone else's.

One more definitional choice: where the stopwatch sits. This capture's client and model shared a machine, so its stamps contain no network. A client across the internet folds the network into TTFT and E2E and its jitter into the gaps; a probe inside the server sees none of it. The two placements measure different things: where your users sit gives the number they feel, and next to the server isolates the machine. Either way, say where the stopwatch stood.

Everything here came from the gentlest interrogation a server can face: one client, one request at a time, a line that never formed. The four names describe this server at its calmest. Real serving is many clients at once, and a benchmark has to decide how to apply that pressure, a choice that changes every number this lesson defined, TTFT most of all. How a benchmark offers load is the next lesson.