Goodput vs throughput in LLM inference

Goodput vs throughput in LLM inference is the difference between work the server finishes and work the user can accept. A request that returns late still raises throughput. It contributes nothing to goodput.

That distinction can reverse a capacity decision. The serving plan with more tokens per second may deliver fewer requests inside the time-to-first-token and streaming-latency limits.

The useful comparison is requests that meet every latency objective per second. Once that number is known, cost per acceptable request is one more division.

Goodput vs throughput in LLM inference

Throughput measures all completed work over time. Depending on the benchmark, that may mean output tokens per second, total tokens per second, or requests per second.

Goodput adds a quality gate. NVIDIA AIPerf defines it as completed requests per second that meet specified metric constraints.

For a 60-second run, request throughput is completed requests divided by 60. Request goodput is requests meeting every SLO divided by 60.

The word is not perfectly standardized. Anyscale's metrics guide reports goodput as the percentage of requests meeting all SLOs.

AIPerf calls that percentage good_request_fraction and reserves goodput for the rate. Record both values so a benchmark report cannot hide the denominator.

If 540 of 600 completed requests meet every limit, the attainment fraction is 90 percent and goodput is 9 requests per second over a 60-second window.

Work through two serving plans

Suppose two configurations serve the same prompts for 60 seconds. Every response contains 300 output tokens on average. The SLO requires TTFT at or below 500 ms and TPOT at or below 20 ms.

Plan A completes 720 requests. Its throughput is 12 requests per second and 3,600 output tokens per second. Only 45 percent of requests meet both latency limits.

That leaves 324 acceptable requests. Plan A's goodput is 324 divided by 60, or 5.4 requests per second.

Plan B completes 600 requests. Its throughput is 10 requests per second and 3,000 output tokens per second. Ninety percent meet both limits.

That leaves 540 acceptable requests. Plan B's goodput is 9 requests per second.

Raw throughput selects Plan A. Goodput selects Plan B by 9 divided by 5.4, or 1.67 times. The slower-looking plan supplies 67 percent more usable capacity.

The numbers are hypothetical, but measured systems show the same inversion.

A CNCF serving study found nearly tied request rates with a 220 ms gap in p95 TPOT.

It also found a configuration with about 50 percent more combined token throughput but almost ten times the p95 TPOT. Which plan wins depends on the latency promise, not the tallest throughput bar.

Count a request only when every limit passes

Interactive LLM requests expose at least two distinct waits. TTFT covers queuing and prefill before the first token. TPOT covers the average pace of later tokens.

Fanout's prefill and decode guide explains why one configuration can improve the first wait while worsening the second.

A request with 300 ms TTFT and 28 ms TPOT fails the example SLO. So does one with 700 ms TTFT and 12 ms TPOT. Passing one constraint cannot cancel a miss on another.

Errors belong in the failed set too. AIPerf divides good requests by completed requests plus error requests when it calculates good_request_fraction.

That prevents a server from looking compliant after shedding its hardest requests. Report the attempted rate, completion rate, error rate, goodput, and attainment fraction together.

Use percentile targets when the product promise is percentile based. The DistServe paper defines goodput as the maximum arrival rate that keeps its TTFT and TPOT percentile objectives.

That is a different experiment from filtering individual requests after one fixed-rate run. Write the definition next to the result before comparing two tools.

Turn goodput into a cost number

Assume each plan uses one GPU that costs $4 per hour. The hourly cost is the same, but the acceptable work is not.

Plan A produces 5.4 good requests per second, or 19,440 per hour. Its cost per good request is $4 divided by 19,440, about $0.000206.

Plan B produces 9 good requests per second, or 32,400 per hour. Its cost per good request is about $0.000123.

Plan B costs 40 percent less per acceptable request. A cost dashboard built from raw output tokens would recommend the opposite configuration.

This calculation also exposes an SLO that is too loose. If almost every configuration passes, goodput collapses back toward throughput and stops distinguishing user experience.

An impossible SLO is equally unhelpful. Every plan approaches zero goodput, so the benchmark cannot tell a tuning problem from a hardware limit.

Choose the latency limits from the product path first. Then measure the capacity and cost that remain inside them.

Match the SLO to the workload

Chat usually needs a bounded TTFT and a smooth token stream. An agent may care more about end-to-end latency because the next tool call cannot begin until the full answer arrives.

Offline generation may not need goodput at all. If no user waits and the deadline is measured in hours, aggregate token throughput and cost per token can be the honest targets.

Prompt and output lengths must stay fixed across plans. Request throughput rises when a benchmark silently shifts toward shorter responses, even if the serving system did not improve.

Use a trace or a controlled length distribution that resembles production. Fanout's AI inference engineering guide covers the other workload variables that can invalidate a comparison.

Separate cohorts when their promises differ. A coding completion with a 200 ms TTFT target should not share one goodput number with a long research answer allowed to wait two seconds.

Goodput is conditional on the workload, SLO, arrival process, hardware, model, and engine version. Strip away any of those labels and the number becomes hard to reproduce.

Sweep load instead of testing one point

One concurrency setting shows a point, not capacity. Start below saturation and raise the offered request rate in steps while keeping the prompt distribution constant.

At low load, throughput and goodput should be close. As queues form, throughput may flatten while TTFT rises and goodput falls.

The useful operating point is the highest sustained arrival rate that still meets the required attainment fraction. Leave headroom for traffic variance rather than running at the first cliff.

Do not use only averages. A mean TTFT below 500 ms can coexist with a p95 far above it. Store the per-request measurements needed to evaluate the actual percentile promise.

Warm the model before collecting results, run each point long enough to cross batching cycles, and repeat the sweep. One lucky batch should not choose a deployment.

If the winning point changes with the traffic mix, that is information. It may justify separate pools, request classes, or disaggregated prefill and decode.

Fanout's disaggregated serving calculation shows how independent phase capacity can raise goodput even after KV transfer is charged.

A compact decision rule

Start with the user-facing SLO. For streaming chat, that usually includes both TTFT and TPOT. For a chained agent, add or substitute end-to-end latency.

Run the same trace against every serving plan. For each request, mark success only if the response completes without error and every applicable limit passes.

Calculate request throughput, token throughput, goodput, and good_request_fraction. Then divide hourly infrastructure cost by goodput times 3,600.

Pick the plan with the lowest cost per good request at the required attainment level. Raw throughput remains useful for diagnosing the machine, but it does not make the product decision.

Plan A led the worked example by 3,600 to 3,000 tokens per second. Plan B delivered 9 acceptable requests per second instead of 5.4, which is the capacity users could actually consume.