Goodput vs SLO attainment in LLM serving

Goodput vs SLO attainment is a rate versus a fraction. Goodput counts acceptable requests per second. SLO attainment asks what fraction of the measured requests were acceptable.

They share a numerator but answer different questions. A server can reach 100 percent attainment by accepting almost no traffic, while another can deliver far more acceptable work at 90 percent.

Under one clean measurement window, goodput equals completion throughput multiplied by SLO attainment. Most confusion starts when a report changes the window or the denominator.

Goodput vs SLO attainment uses two denominators

Let N be all completed requests in a 60-second run. Let M be the requests that completed without error and met every stated latency objective.

SLO attainment is M divided by N. Completion throughput is N divided by 60 seconds. Goodput is M divided by 60 seconds.

Multiplying the first two equations gives goodput: completion throughput times SLO attainment.

The paper that revisits SLO and goodput metrics uses the same distinction. It defines attainment as a proportion and goodput as completed, SLO-meeting requests per second.

Fanout's goodput vs throughput guide covers the throughput side. The missing piece is why the percentage cannot replace the rate.

Work through the same 60-second run

Suppose a server completes 600 requests in one minute. Of those, 540 meet both the time-to-first-token and time-per-output-token limits.

Completion throughput is 600 divided by 60, or 10 requests per second. SLO attainment is 540 divided by 600, or 90 percent.

Goodput is 540 divided by 60, or 9 requests per second. The product check gives the same result: 10 times 0.90 equals 9.

Now halve the offered traffic. The server completes 300 requests and all 300 meet the SLO.

Attainment improved from 90 to 100 percent. Goodput fell from 9 to 5 requests per second.

The light-load run is more reliable per request but supplies 44 percent less acceptable capacity.

SLO attainment can reward an idle server

Attainment is useful when the arrival rate is fixed by real traffic or by a controlled benchmark point. Without that context, it rewards low load.

A one-GPU server at one request per second may post 100 percent attainment. The same server at ten requests per second may post 90 percent.

If production needs eight acceptable requests per second, the first result is irrelevant even though its percentage is perfect. Its maximum possible goodput is only one.

This is why capacity studies sweep the offered rate. The DistServe paper defines goodput as the maximum arrival rate that can sustain its percentile SLO targets.

That definition bakes an attainment requirement into the capacity search. It does not mean the words goodput and attainment are synonyms.

The sweep asks for the highest load that keeps the chosen fraction inside the latency limits. Attainment is the constraint; goodput is the useful rate left under it.

Goodput can hide a bad fraction

Two systems can report the same goodput while giving requests very different odds of success.

System A completes 50 requests per second and every request passes. Its goodput is 50 requests per second and its attainment is 100 percent.

System B completes 100 requests per second but only half pass. Its goodput is also 50 requests per second, while its attainment is 50 percent.

A bulk job that retries failures might tolerate System B. Interactive chat usually will not. Half the users experience a miss even though aggregate acceptable capacity matches System A.

Report the pair. Goodput tells the fleet planner how much acceptable work arrives. Attainment tells the product owner how likely one request is to receive that service.

Raw request throughput belongs beside both. Without it, the reader cannot verify the identity or see how much work was filtered out.

Denominators decide whether the identity holds

The simple formula assumes N means the same thing in all three metrics. Real tools may count attempted, admitted, completed, or successful requests.

NVIDIA AIPerf counts completed requests that meet every configured constraint for goodput.

Its good-request fraction includes error requests in the denominator. That choice stops a server from improving the percentage by failing hard requests before they complete.

Suppose 1,000 requests arrive. The gateway rejects 100, the engine errors on 50, and 765 of the remaining 850 completed requests meet every latency limit.

Completed-only attainment is 765 divided by 850, or 90 percent. Attempt-based attainment is 765 divided by 1,000, or 76.5 percent.

Both calculations can be internally consistent. Only the second describes the chance that an arriving user receives an acceptable answer.

Write the numerator and denominator next to every percentage. Also state whether cancellations, timeouts, admission rejections, and client disconnects count as failures.

Per-request limits and percentile SLOs are different tests

AIPerf can mark each request against constraints such as TTFT below 500 ms and inter-token latency below 50 ms. One request either passes every constraint or it does not.

A percentile SLO asks a fleet-level question, such as whether 99 percent of requests have TTFT below 500 ms over a window.

These tests can produce different goodput numbers. Filtering requests after a fixed run is not the same as finding the maximum arrival rate that keeps a percentile target.

Keep TTFT and decode pace separate. Fanout's prefill and decode guide explains why the two phases fail under different load patterns.

A request that passes TTFT but misses TPOT is not fully acceptable when both are in the contract. Passing one limit cannot offset missing another.

The workload matters too. Prompt length, output length, cache hits, and arrival bursts change the queue, so the same server has different attainment and goodput on different traces.

Binary goodput has a known blind spot

Ordinary goodput assigns zero value to a request once it misses any SLO. That makes the metric easy to calculate but creates a bad optimization incentive.

The revisiting paper shows that a scheduler can improve measured goodput by abandoning requests that already missed a deadline. Killing them frees resources for requests that can still pass.

Users may prefer a late complete answer to an aborted stream. The metric does not express that preference because both a small miss and a total failure contribute zero.

The paper proposes smooth goodput, which assigns partial benefit and penalizes user idle time. It is useful research, but it adds parameters for reading speed and waiting cost.

For a normal capacity benchmark, binary goodput remains understandable. Pair it with completion rate, error rate, and latency distributions so the scheduler cannot win by discarding work.

How to report an LLM serving result

Start with one explicit measurement window and one production-like request trace. State the offered arrival rate before reporting what the server completed.

Report attempted requests, admitted requests, successful completions, errors, and SLO-meeting completions. These counts make every later metric reproducible.

Then report request throughput, SLO attainment, and goodput with units. Add TTFT and TPOT percentiles rather than only their means.

Sweep the offered rate until attainment crosses the product target. The last stable point is the capacity result, with headroom still needed for traffic variance.

Fanout's disaggregated serving calculation shows how separate prefill and decode pools can change that capacity boundary.

Goodput vs SLO attainment is not a choice between metrics. Use the rate for acceptable capacity, the fraction for reliability, and the raw counts to prove both came from the same denominator.