--- title: "User idle latency in LLM serving" description: "Turn reading speed and buffered tokens into a user-visible stall deadline, then measure it separately from TPOT and server-side inter-token latency." canonical_url: "https://fanout.sh/blog/user-idle-latency-llm-serving" md_url: "https://fanout.sh/blog/user-idle-latency-llm-serving.md" last_updated: "2026-09-28" access: "public" --- # User idle latency in LLM serving Turn reading speed and buffered tokens into a user-visible stall deadline, then measure it separately from TPOT and server-side inter-token latency. - Author: Suraj Gaud - Published: 2026-09-28 - Track: Inference engineering - Access: Public - Tags: LLM serving, user idle latency, token streaming, inter-token latency, TPOT, buffering, scheduling User idle latency in LLM serving is not the same as the gap between two server emissions. A client can hide a long generation pause when it has enough tokens buffered ahead of the reader. The deadline is simple. If the buffer holds B unread tokens and the user consumes r tokens per second, a server pause stays invisible for at most B divided by r seconds. This converts reading speed into a scheduling budget that average TPOT cannot provide. ## User idle latency in LLM serving Serving dashboards usually begin with time to first token, time per output token, and end-to-end latency. Those metrics describe the request at the server or transport boundary. The user experiences a different timeline: when rendered text becomes available faster or slower than they consume it. The[Andes paper](https://arxiv.org/abs/2404.16283)models text streaming around an expected token-delivery speed and the actual timeline at which a user can digest tokens. Its client buffer can withhold tokens generated ahead of that pace. The buffer then drains while the server pauses or serves someone else. Call a user idle interval any time the reader is ready for the next token but the display buffer is empty. This definition excludes a server gap that occurs while unread text is still on screen. It also excludes the initial wait before any token appears. TTFT remains a separate product constraint because there is no earlier response text to consume. Fanout's[smooth goodput guide](/blog/smooth-goodput-vs-goodput)separates average decode speed from the cadence of individual token gaps. User idle latency adds client consumption to that cadence. ## Derive the hidden-stall deadline Let B be the number of generated tokens that the user has not consumed. Let r be the estimated consumption rate in tokens per second. With no new generation, the buffer lasts B / r seconds. That is the maximum invisible pause from the current state. For a user consuming 6 tokens per second, an 18-token buffer hides 3 seconds. A 2.5-second server pause causes no visible idle time; a 4-second pause exposes roughly 1 second. If the product wants J seconds of jitter protection, the target buffer is r times J tokens. At 6 tokens per second and J = 2, keep about 12 unread tokens available. Tokens are not words, so r must come from the tokenizer and display policy used by the application. A fixed words-per-minute assumption is only a starting estimate. The rate can also change by modality. Voice playback, code completion, translation, and long-form reading do not consume generated tokens at the same pace. Treat B / r as a live deadline, not a universal SLO. Every displayed token reduces B, while every newly generated token increases it. ## Average TPOT can hide the bad gap The current[vLLM metrics documentation](https://docs.vllm.ai/en/latest/design/metrics/)distinguishes request-level TPOT from inter-token latency. TPOT is recorded once per finished request as decode duration divided by the number of output gaps. Inter-token latency records the wall-clock gap for each streamed output event. Consider a response with 20 output tokens after the first token, so there are 19 measured gaps. Suppose the total time across those gaps is 0.95 seconds. Uniform 50-millisecond gaps give a TPOT of 50 milliseconds. A second stream with eighteen gaps near 2.8 milliseconds and one 900-millisecond gap has almost the same TPOT. The first stream looks steady. The second contains a visible freeze unless earlier tokens created enough buffer to cover 900 milliseconds of consumption. A request-level average cannot identify which stream occurred. The ITL histogram can expose the 900-millisecond event, but it still does not say whether the reader ran out of text. Server cadence and user idle are therefore related but not interchangeable. Measure both when the application paces or buffers output. ## The client buffer changes what the user sees Without pacing, the client may render every event as soon as it arrives. In that design, network and server jitter appear directly in the visual stream. With pacing, the client displays at an expected rate and stores any surplus. Bursty generation can then look smooth even though the arrival trace remains uneven. Andes uses this buffer to let the scheduler preempt a request that has enough tokens in reserve. The server can work on a more urgent request while the first user's buffer drains. This does not make latency disappear. It moves work earlier, stores the result, and spends that inventory during a later pause. Client telemetry must preserve both timelines. Record when each token or chunk arrives and when it becomes visible or audible to the user. Also record buffer occupancy at each scheduling decision. Arrival timestamps alone cannot reconstruct the display policy if the pacing rate changes during a request. A transport event may bundle multiple accepted tokens under speculative decoding. Count buffered tokens, not only event callbacks, when computing B. ## Prefill interference creates decode stalls LLM requests alternate between compute-heavy prefill work and memory-heavy decode steps. A large prefill can delay decode iterations for requests already streaming. [Sarathi-Serve](https://arxiv.org/abs/2403.02310)addresses this interference with chunked prefills and stall-free schedules that attach decode work to more uniform batches. [DistServe](https://arxiv.org/abs/2401.09670)separates prefill and decode onto different GPUs so the two phases can use independent resource plans and latency targets. Both systems treat smooth decode latency as a serving concern, but a client buffer changes how strict each request's immediate decode deadline is. A request with 30 unread tokens can tolerate a longer scheduling pause than one with 2 unread tokens, even when both have the same model, prompt length, and average TPOT target. That does not remove the need to control ITL. A scheduler that repeatedly drains buffers to zero creates visible stalls and leaves no room for network jitter. The buffer equation supplies a per-request urgency signal. Chunking and disaggregation supply ways to meet the resulting deadlines. ## A two-user scheduling example User A consumes 6 tokens per second and has 24 unread tokens. Its current invisible-pause budget is 4 seconds. User B consumes at the same rate but has only 3 unread tokens. Its budget is 0.5 seconds. If a 1.2-second scheduling pause is unavoidable, pausing A is invisible. Pausing B creates about 0.7 seconds of user idle latency. Now suppose the server can generate 12 tokens for B at 24 tokens per second. Generation takes 0.5 seconds while B consumes about 3 tokens. B started with 3 tokens, gained 12, and consumed 3. It ends with about 12 unread tokens, enough for a 2-second pause at the same reading rate. The scheduler has converted half a second of GPU service into two seconds of buffer coverage. It can now switch to other work without immediately freezing B's display. This arithmetic is workload-specific. The generation rate may fall as batch size grows, and a token bundle may arrive later than its compute completion time. Use measured arrival rate and client consumption, not model peak throughput, when estimating the new buffer. ## Build SLOs from the consumption timeline Keep TTFT as the initial responsiveness SLO. After the first token, define a visible-idle budget over the display timeline. One useful target is the fraction of requests with no idle interval longer than a chosen threshold after streaming begins. Another is total visible idle time per response. Two 300-millisecond gaps and one 600-millisecond gap have the same total, but may have different product impact. Track both maximum gap and cumulative idle when that distinction matters. Do not reduce the entire experience to one average. At each server decision, estimate time to buffer exhaustion as B / r. Subtract transport and render safety margins to obtain the latest safe delivery time. If the estimate is 1.5 seconds and the safety margin is 0.3 seconds, schedule enough new output to arrive within about 1.2 seconds. The margin should include observed network tail latency, event-loop delay, and client pacing uncertainty. It should not be copied from a server-only benchmark. Fanout's[goodput versus SLO attainment guide](/blog/goodput-vs-slo-attainment)explains why useful capacity depends on meeting request constraints, not only producing more tokens. Consumption-aware deadlines make the constraint closer to what the user can observe. ## Instrument three separate layers At the engine layer, collect TTFT, request TPOT, per-event ITL, queue time, batch size, and prefill/decode timing. At the transport layer, timestamp each streamed event and count the tokens carried by it. This catches proxy buffering, chunk coalescing, and network delay. At the client layer, record display time, estimated consumption rate, unread-token count, and every interval where the buffer reaches zero after the first token. Join the layers with a request identifier and monotonic timestamps. Wall clocks across machines can add false gaps if they are not synchronized. Compare server ITL p95 and p99 with user-idle p95 and p99. A high ITL tail with low user idle means buffering is working. A low server ITL tail with high user idle points toward transport buffering, client rendering, a bad consumption estimate, or a buffer that never had time to build. The[prefill versus decode guide](/blog/prefill-vs-decode-llm-inference)helps attribute an engine-side stall to the correct phase before changing scheduler policy. ## Do not buy smoothness with a slow start A client could wait for a large fixed buffer before showing any text. That reduces later stalls by increasing TTFT, which is often the wrong trade for chat. Show the first token promptly, then build a modest reserve when generation is faster than consumption. The reserve can grow without delaying the initial response. Cap the buffer so the server does not generate far ahead of a user who may cancel, edit, or abandon the request. Excess inventory spends GPU time and memory on text that may never be consumed. Make the cap product-specific. Code completion may need only a small reserve because the user can accept or reject quickly, while narrated output may benefit from a steadier queue. When generation falls below consumption for a sustained period, no finite buffer can prevent eventual idle. The system must improve decode capacity, reduce load, or lower the delivery rate. [Disaggregated prefill and decode](/blog/disaggregated-prefill-and-decode)can reduce one source of interference, but the client still needs direct measurement of visible stalls. In operation, schedule against buffer exhaustion and verify against the display timeline. B divided by r tells you how long the user can wait without knowing they waited. --- This representation contains public Fanout content only. Protected Pro lessons, account data, billing, checkout, and pricing are not included. Browse the public content map: https://fanout.sh/sitemap.md