--- title: "P95 ITL vs average TPOT" description: "See why p95 inter-token latency and average TPOT can disagree, with worked streams, weighting rules, and the right metric for each serving SLO." canonical_url: "https://fanout.sh/blog/p95-itl-vs-average-tpot" md_url: "https://fanout.sh/blog/p95-itl-vs-average-tpot.md" last_updated: "2026-10-01" access: "public" --- # P95 ITL vs average TPOT See why p95 inter-token latency and average TPOT can disagree, with worked streams, weighting rules, and the right metric for each serving SLO. - Author: Suraj Gaud - Published: 2026-10-01 - Track: Inference engineering - Access: Public - Tags: inter-token latency, TPOT, tail latency, LLM serving, benchmarking, streaming, SLO P95 inter-token latency and average TPOT answer different questions. Average TPOT compresses each request into one number. P95 ITL looks at the slow end of the gaps observed while responses stream. That difference matters when a response is usually fast but occasionally pauses. The same average TPOT can describe a smooth stream or one that stops every few tokens. The comparison is only useful after naming the samples behind each statistic. A percentile without its population is not a complete metric. ## P95 ITL vs average TPOT starts with two populations For one request, TPOT equals generation time after the first token divided by output tokens minus one. If every streamed event contains one token, that value is also the mean of the request's individual inter-token gaps. The current[vLLM benchmark guide](https://github.com/vllm-project/vllm/blob/main/docs/benchmarking/cli.md)keeps two collections. It calculates one TPOT per successful request and collects individual ITL gaps across all successful requests. Mean TPOT therefore averages requests. Mean ITL averages observed gaps. A long answer contributes one TPOT value but many ITL values. P95 ITL sorts the collected gaps and finds the tail cutoff. P95 TPOT sorts the per-request averages instead. These percentiles can move in opposite directions on the same trace. NVIDIA's[benchmark metric definition](https://docs.nvidia.com/nim/benchmarking/llm/latest/metrics.html)uses ITL for the average decode interval and warns that tools differ on whether they include the first-token wait. Before comparing reports, check whether "ITL p95" means a percentile of token gaps, request-level average gaps, or server histogram buckets. The label alone does not settle it. ## Equal average TPOT can hide a stalled stream Consider two responses with 101 output tokens. Each has 100 gaps after the first token. Stream A has 100 gaps of 20 milliseconds. Generation takes 2,000 milliseconds, so average TPOT is 20 milliseconds. P95 ITL is also 20 milliseconds. Stream B has 90 gaps of 10 milliseconds and 10 gaps of 110 milliseconds. Its generation time is still 2,000 milliseconds, so its average TPOT is also 20 milliseconds. The user does not see the same stream. Stream B pauses for 110 milliseconds ten times. Under the nearest-rank rule, its p95 ITL is 110 milliseconds. An average TPOT dashboard says the two streams tie. The gap distribution says one is smooth and one is bursty. TPOT still measures useful token progress over the whole decode. It is the right denominator for many throughput and request-level SLO calculations. The example shows why a mean cannot certify smooth delivery. Add at least one tail statistic or a deadline miss rate when pauses affect the product. Fanout's[smooth goodput comparison](/blog/smooth-goodput-vs-goodput)treats token deadlines as graded value rather than a single request pass or fail. That view is useful when a few late gaps matter before the response ends. ## Request weighting changes the aggregate Now compare one 101-token answer at 10 milliseconds per gap with one two-token answer whose only gap is 110 milliseconds. Request-weighted average TPOT is 60 milliseconds: one request contributes 10 and the other contributes 110. Gap-weighted mean ITL is about 11 milliseconds: 100 fast gaps and one slow gap contribute to the pool. Neither value is wrong. Request weighting says both users count equally. Gap weighting says every observed streaming interval counts equally. The short request dominates request-weighted TPOT even though it produced one decode gap. The long request dominates pooled ITL because it produced 100 gaps. Anyscale's[ranked latency guide](https://docs.anyscale.com/llm/serving/benchmarking/metrics)documents this distinction for a vLLM-backed service. It recommends request-weighted TPOT for per-request behavior and token-weighted ITL for steady streaming speed. For an interactive product, keep both populations. A global ITL percentile can hide a small class of short, bad requests. A request percentile can hide repeated stalls inside otherwise acceptable request averages. Break the distributions down by output length, prompt length, route, model, and offered load. A single fleet-wide p95 can still mix unlike workloads. ## Streamed chunks can separate ITL from token progress One streamed event does not always equal one token. Speculative decoding can accept several draft tokens in one engine step and send them together. The vLLM guide gives a five-token example with two observed 40-millisecond ITL samples. Mean ITL is 40 milliseconds because the client saw two event gaps. TPOT is 20 milliseconds per token because 80 milliseconds of generation time produced four tokens after the first. The two values now disagree even inside one request. ITL measures the cadence of output events. TPOT measures elapsed decode time per output token. This distinction also appears when a gateway buffers several token deltas before flushing them. A smooth server can look bursty at the client, or a bursty server can be hidden behind buffered chunks. Measure at the surface named by the SLO. A server-side metric is useful for engine diagnosis. A client-side event gap is closer to what the user sees. The[AIPerf metrics reference](https://github.com/ai-dynamo/aiperf/blob/main/docs/metrics-reference.md)separates request records from aggregate metrics and preserves per-request exports for custom analysis. Keep event timestamps or raw gap samples when possible. Once a run is reduced to mean TPOT and p95 ITL, you cannot reconstruct which requests stalled or whether chunks carried multiple tokens. ## Percentile method and sample size can move p95 Percentile libraries interpolate differently. With 20 gaps, the 95th percentile sits at the boundary between the largest values, so two tools can report different numbers from the same list. The problem is larger with small or bucketed samples. A Prometheus histogram can only locate a percentile within its configured buckets. A client trace with exact timestamps has finer resolution. Record the percentile method, sample count, units, warm-up policy, and measurement point. Compare the same choices between runs. Do not merge warm-up, steady state, and overload recovery into one distribution unless that mix matches production traffic. A low p95 can coexist with a bad five-second incident if the benchmark runs for an hour. Report p50, p95, p99, maximum, and the fraction of gaps above the product deadline. The deadline miss rate often explains the result more directly than another percentile. For a 50-millisecond streaming deadline, "2.3% of gaps exceeded 50 milliseconds" is operationally clearer than p95 alone. It also avoids pretending that the 95th percentile is a universal boundary. ## Match the metric to the failure Use average TPOT to compare request-level decode efficiency when output lengths and request weighting are controlled. Use p95 or p99 ITL to find pauses that a mean hides. Pair it with a histogram or threshold exceedance rate so one percentile does not conceal the rest of the tail. Use p95 TPOT when the contract applies to each request's average token progress. That is different from requiring 95% of individual gaps to stay below a limit. Use end-to-end latency when downstream work cannot start until the response finishes. A streaming metric can look healthy while a long answer still misses its completion deadline. Use goodput when capacity only counts if requests satisfy several objectives. Fanout's[goodput and SLO guide](/blog/goodput-vs-slo-attainment)shows why the SLO predicate must be explicit before throughput has meaning. For production diagnosis, plot gap tails against queue depth, batch size, prefill work, preemptions, and output chunk size. A p95 spike says where the symptom is, not which scheduler action caused it. ## A benchmark report that can be audited Start with client timestamps for request send, first output, every streamed output, and final output. Store the token count attached to each event. For every request, calculate TTFT, generation time, output token count, TPOT, mean ITL, maximum ITL, and count of gaps above the deadline. Across requests, publish the distribution of request TPOT values. Separately publish the distribution of individual client-observed gaps. Name the weighting. State whether long responses supply more ITL samples and whether each request supplies exactly one TPOT sample. Repeat the run across offered load rather than quoting one concurrency point. Tail gaps often rise near saturation before average TPOT makes the change obvious. Save the raw records beside the summary. A future SLO may ask for p97, a 75-millisecond deadline, or a route-specific view that the original report did not compute. P95 ITL and average TPOT belong next to each other. One exposes slow streaming intervals. The other measures average token progress. They only conflict when the report hides what was averaged. --- This representation contains public Fanout content only. Protected Pro lessons, account data, billing, checkout, and pricing are not included. Browse the public content map: https://fanout.sh/sitemap.md