Smooth goodput vs goodput in LLM serving

Smooth goodput vs goodput asks whether a late request is worth zero. Standard goodput applies a binary SLO gate. Smooth goodput gives generated output value, then subtracts a penalty for the time a user had nothing left to read.

The difference matters after a request crosses its deadline. Binary goodput can reward dropping that request because it already counts as zero. Smooth goodput still values useful output, so abandoning the stream destroys value.

Smooth goodput is not a drop-in universal metric. Its reading rate and penalty function encode product choices. Run it beside ordinary goodput until those choices match observed user behavior.

Smooth goodput vs goodput

Goodput in LLM serving usually means completed work that satisfies latency objectives. A request may need to meet a TTFT limit and a TPOT or TBT limit before its output counts.

Definitions vary. Some systems report requests per second at a target SLO attainment. The DistServe paper asks for the maximum per-GPU request rate while at least 90% of requests meet latency goals.

The smooth goodput paper writes goodput as SLO-qualified output tokens divided by an observation window. Both forms use the same binary indicator: a request passes or it contributes zero.

Binary scoring is easy to explain and compare. It answers a real capacity question: how much traffic can the system finish inside a declared service boundary?

It also creates a cliff. A response that misses by 20 milliseconds receives the same zero as a response that stalls for 20 seconds.

Fanout's goodput vs SLO attainment guide separates the qualified work rate from the fraction of requests that qualify.

Binary goodput rewards giving up

Suppose a scheduler knows that one active request has already missed its token deadline. Finishing it cannot restore its contribution to binary goodput.

Dropping the request frees KV-cache space and decode slots for requests that can still pass. The goodput counter may rise even though one user receives an incomplete answer.

The smooth goodput authors use this as their second counterexample. Once a request has zero binary value, a goodput-maximizing scheduler has a reason to kill it.

This is a metric failure, not proof that admission control is wrong. Rejecting work before it starts can protect a service. Silently abandoning a partly delivered answer is a different user outcome.

Binary goodput also hides distance from the threshold. It cannot distinguish a near miss that remains useful from an unusable stream with repeated empty waits.

The goodput vs throughput explainer shows why qualifying work is better than counting every token. Smooth goodput asks how to value the work that falls just outside that gate.

User idle latency follows the reading buffer

TBT measures the gap between adjacent tokens. A long gap does not always make a user wait because earlier tokens may still be unread.

The paper gives a simple example. If ten tokens arrive during the first second and the user reads four tokens per second, a later one-second generation stall can remain hidden behind the buffered text.

If only two tokens arrive before the same stall, the user runs out of text and waits. TBT sees the same one-second gap in both cases, while the experience differs.

Smooth goodput assigns a deadline to every token based on an assumed information-consumption rate. For a rate of q tokens per second, token i has a relative deadline of i / q seconds.

If token i arrives at time t_i, its lateness is t_i minus that deadline. User idle latency is the maximum lateness across the stream, clamped to zero when every token stays ahead of consumption.

This maximum keeps later bursts from erasing an earlier wait. Once a user sat idle for 800 milliseconds, a fast final paragraph does not give that time back.

The rate should differ by workload. Code, equations, prose, audio, and tool traces are consumed differently. One global tokens-per-second assumption will rank some streams incorrectly.

Worked smooth goodput arithmetic

Consider a ten-second window with three requests that each generate 100 tokens. Request A stays inside every deadline. Request B reaches a maximum user idle latency of 0.2 seconds. Request C reaches 2 seconds.

Under binary goodput, assume only A passes the SLO. The qualified output is 100 tokens, so goodput is 100 / 10 = 10 SLO-qualified tokens per second.

B and C contribute zero even though both produced complete 100-token answers. The metric records no difference between their short and long waits.

For an illustrative smooth score, set alpha to 100 token-value units. Let the penalty fraction be f(l) = min(l / 2 seconds, 1). These are example choices, not defaults from the paper.

A has zero idle latency, so its benefit is 100 - 100 x 0 = 100. B has a penalty fraction of 0.1, so its benefit is 100 - 100 x 0.1 = 90.

C reaches the two-second cap. Its benefit is 100 - 100 x 1 = 0. Total benefit is 190, and smooth goodput is 190 / 10 = 19 benefit units per second.

The 19 is not directly comparable to 10 unless the team accepts this benefit model. Its value is the ordering: A is worth more than B, B is worth more than C, and finishing B still helps.

Change alpha or the penalty curve and the ordering can change. That sensitivity is a feature only when the parameters come from a defensible product decision.

Smooth goodput also fixes an output-delay trick

TBT deadlines are often relative to the previous delivered token. That lets a server improve measured TBT by holding an already generated token until the threshold.

The delayed token moves the next deadline later. The metric improves even though every affected token reaches the user at the same time or later.

The paper avoids this by making each deadline relative to the request start and a consumption rate. Delivering a token early builds a buffer instead of tightening the next token's deadline.

This changes how chunked prefill looks. Breaking a long prefill into chunks can reduce the largest adjacent-token gap, but it may add repeated KV-cache reads and increase absolute latency.

Smooth goodput asks whether the user ran out of readable output. That test can favor chunking when it prevents idle time and reject it when it only makes TBT look regular.

Fanout's chunked prefill guide explains the scheduling mechanism. The metric should evaluate that mechanism, not become its target by assumption.

The penalty function is a product policy

The paper defines request benefit as generated tokens minus alpha times a function of user idle latency. Smooth goodput sums those benefits and divides by the serving interval.

Alpha controls how much the service values waiting time relative to output. The function f controls whether the penalty is linear, capped, or steep near a product-specific boundary.

A linear penalty treats each extra second equally. A convex penalty can make long stalls disproportionately expensive. A capped penalty prevents one pathological request from dominating a whole window.

Each option embeds a judgment. None can be chosen from GPU traces alone.

Use cancellation, regeneration, session abandonment, complaint, or task-completion data to calibrate the curve. Keep the raw latency distributions available so the composite score cannot hide a bad tail.

Do not use smooth goodput for offline batch inference where nobody consumes a live stream. End-to-end completion time and cost per accepted output are more direct there.

How to measure it without losing the basics

Record the request arrival time and the delivery time of every streamed token. Generation timestamps are not enough if a proxy buffers output before the client receives it.

Choose a consumption model for each request class. Convert its rate into token deadlines, then calculate maximum lateness and the request benefit.

Report throughput, TTFT, TBT or inter-token latency, binary goodput, SLO attainment, and smooth goodput together. A composite score should add a view, not remove the measurements needed to debug it.

Plot all metrics against offered load. The smooth-goodput paper found a peak that exposed a balance between generated value and user waiting, while throughput continued toward saturation.

Run the calculation on completed requests before allowing it to drive scheduling. Check whether the ranking agrees with real cancellations and task completion across short chat, long reasoning, and code workloads.

Finally, test the incentive directly. Compare a scheduler that finishes late requests with one that abandons them. A metric intended to represent users should not call the second policy better without evidence.

When each metric should decide

Use binary goodput for a contractual capacity claim. It is well suited to statements such as requests per second while 99% meet a defined TTFT and TPOT boundary.

Use smooth goodput to compare scheduling policies inside that capacity envelope, especially when requests stream for a long time and near misses still carry value.

Keep both when the service is young. Binary goodput supplies a stable operating boundary. Smooth goodput reveals cliffs, partial value, and incentives that the boundary discards.

Goodput counts work that passed. Smooth goodput models what the service should do after some work did not.