--- title: "LLM token buffer occupancy scheduling" description: "Turn unread streamed tokens into scheduling runway, then account for consumption rate, preemption cost, KV memory, and overload fallback." canonical_url: "https://fanout.sh/blog/llm-token-buffer-occupancy-scheduling" md_url: "https://fanout.sh/blog/llm-token-buffer-occupancy-scheduling.md" last_updated: "2026-10-06" access: "public" --- # LLM token buffer occupancy scheduling Turn unread streamed tokens into scheduling runway, then account for consumption rate, preemption cost, KV memory, and overload fallback. - Author: Suraj Gaud - Published: 2026-10-06 - Track: Inference engineering - Access: Public - Tags: LLM serving, token buffer, scheduling, preemption, KV cache, streaming, TokenFlow LLM token buffer occupancy scheduling treats unread streamed tokens as a time budget. A request with 80 buffered tokens and a 20-token-per-second consumer has four seconds of runway. A second request with ten buffered tokens at the same consumption rate has only half a second. Serving the second request first can prevent a visible stall while the first user keeps reading from the buffer. Buffer size alone is insufficient. A safe scheduler must also finish eviction, KV-cache loading, and rescheduling before that runway disappears. ## LLM token buffer occupancy scheduling The[TokenFlow paper](https://arxiv.org/html/2510.02758)starts from a mismatch in interactive serving. Active requests can generate faster than users consume text while newly arrived requests wait for a first token. First-come-first-served scheduling preserves the active batch, but it can spend GPU time producing tokens that sit unread. Prefill-first scheduling can reduce some queueing while disrupting streams already near an empty buffer. TokenFlow tracks a virtual output buffer for every request. It uses unread token count, target consumption rate, waiting time, and memory cost to decide which requests run, pause, or resume. The[project implementation](https://github.com/SJTU-RTEAS/TokenFlow)builds this policy on SGLang and pairs it with proactive GPU-to-CPU KV-cache movement. Fanout's[smooth goodput guide](/blog/smooth-goodput-vs-goodput)explains the related measurement idea: early tokens build a cushion, so one long generation gap may remain invisible to the reader. Buffer-aware scheduling turns that observation into a control policy. ## Convert buffered tokens into runway Let b be unread tokens in the output buffer and r be the required consumption rate in tokens per second. The simplest runway estimate is b divided by r seconds. If b is 60 and r is 15, the request has four seconds before its reader catches the generation frontier. If b falls to 12, the runway is 0.8 seconds. Now estimate the time needed to switch the request out and restore it later. Call the eviction, load, and scheduler times e, l, and s. TokenFlow's admission condition requires b to be at least a safety factor times r times the sum of e, l, and s. Dividing both sides by r makes the logic plain. The buffer runway must exceed the full transition time with headroom. Suppose e is 80 ms, l is 220 ms, s is 20 ms, and the safety factor is 1.5. The protected interval is 1.5 times 0.32 seconds, or 0.48 seconds. At 20 tokens per second, the request needs at least 9.6 unread tokens. Round that requirement up to ten tokens before treating the request as safely preemptible. A fixed token threshold would be wrong here. Ten tokens cover half a second at 20 tokens per second, but only a quarter second at 40 tokens per second. ## A worked three-request schedule The paper gives a toy server with capacity for 40 tokens per second and two concurrent requests. R1 consumes 20 tokens per second, while R2 consumes 30. Both arrive at time zero and start building buffers. R3 arrives two seconds later with a target rate of 25 tokens per second. At that moment, neither active request has enough buffered runway for a safe switch. Admitting R3 immediately would risk starving R1 or R2 while their state moves. By three seconds, R1 has accumulated enough unread output because its consumption target is lower. The scheduler pauses R1, offloads its state, and runs R3. R1 does not appear paused to the user because its buffered tokens continue to drain at 20 tokens per second. At five seconds, R1 approaches depletion. R2 now has the larger cushion, so the scheduler pauses R2 and resumes R1 before the reader runs out. This schedule uses the same generation capacity differently. It moves service toward the request with the least safe runway instead of letting one active stream race far ahead. The example also shows why buffer occupancy must be paired with rate. Thirty unread tokens are comfortable for a 10-token-per-second reader and urgent for a 60-token-per-second speech pipeline. ## Price the context switch before preempting Pausing a request frees compute, but its KV cache still represents every processed prompt and output token. The server must keep that state, move it, or recompute it. Fanout's[KV-cache memory formula](/blog/kv-cache-formula-llm-inference-memory)shows why long contexts make this a memory decision as well as a scheduling decision. TokenFlow estimates both I/O time and recomputation time. Its switching-cost approximation uses the smaller of moving the saved state and rebuilding it with prefill. A short prompt with a slow host-memory path may be cheaper to recompute. A long prompt with available PCIe bandwidth may be cheaper to restore from CPU memory. The choice can change during one run. I/O queue depth, measured transfer bandwidth, prompt length, and recent prefill latency all move the crossover point. Proactive write-through changes the timing. TokenFlow copies newly generated KV entries toward host memory while the request is still running, so a later eviction has less data left to write. The paper reports that hierarchical memory management reduced preemption overhead by 20.3% in its evaluation. That result belongs to the co-designed scheduler and memory path, not to buffer ranking alone. A scheduler that copies the priority formula without the memory path can create frequent stalls from synchronous cache transfers. ## Raw throughput can reward useless lead Raw output throughput counts every generated token equally. It therefore rewards a server that pushes one request hundreds of tokens ahead even while another user waits for the first token. TokenFlow defines token utility from buffer occupancy. Its evaluation fully counts tokens while the buffer stays below 10% of response length, decays their value from 10% to 20%, and excludes tokens beyond 20%. The exact thresholds are an evaluation choice, not a universal human-reading model. Their purpose is to stop excess buffered output from looking as valuable as output a user can consume soon. The paper combines useful-token value with penalties for time to first token and time spent with an empty buffer. This distinguishes a fast start with a mid-stream stall from a genuinely smooth response. In the reported experiments, TokenFlow improved effective throughput by up to 82.5% and reduced P99 time to first token by up to 80.2% without reducing raw token throughput. Those are system-and-workload results, not portable constants. An operator should reproduce them with the actual model, prompt lengths, output rates, host memory, and interconnect. ## Overcommitment needs a fallback Buffering creates room to time-share the GPU, but it cannot create generation capacity. If the working set asks for rates r1 through rn, the sum of those rates must stay below the measured serving capacity. TokenFlow writes this as the sum of request rates being no greater than capacity. When demand crosses that bound, every buffer eventually drains. More aggressive preemption only rearranges which request stalls and adds context-switch work. TokenFlow falls back to first-come-first-served scheduling with memory-aware admission in this condition. It keeps excess requests offloaded until capacity becomes available. The fallback belongs in the design. A production policy needs an explicit overload mode instead of applying normal buffer priorities after their assumptions fail. Capacity also changes with batch size and sequence length. Estimate it from current execution metrics rather than a single offline tokens-per-second number. The[prefill versus decode guide](/blog/prefill-vs-decode-llm-inference)explains why a burst of long prompts can move the capacity boundary even when the model and GPU stay fixed. ## Compare nearby scheduling objectives The[Andes paper](https://arxiv.org/html/2404.16283)also models an ideal consumption timeline and uses token-level preemption to improve quality of experience. Andes shows a failure that average time per output token can hide: a long pause in the middle can coexist with an on-time first and last token. A client-side token pacer exposes the actual consumption timeline. TokenFlow adds an explicit useful-throughput objective and co-designs hierarchical KV-cache movement around repeated preemption. The[PASCAL paper](https://arxiv.org/abs/2602.11530)addresses another adjacent case. It separates hidden reasoning from visible answering and paces answer tokens while prioritizing reasoning work that controls first-token delay. These schedulers optimize different request models. TokenFlow assumes a target output-consumption rate. Andes centers end-to-end consumption quality. PASCAL adds phase information for reasoning models. Do not collapse them into one "lowest buffer first" rule. The right state includes what the user can see, how quickly output is consumed, and what it costs to restore the request. ## Instrument the policy before enabling it Record unread buffer tokens and convert them to seconds of runway with each request's target rate. Measure eviction, load, scheduling, and recomputation latency as distributions. Use tail values in the safety factor because an average transition time can still empty a buffer. Track the I/O queue and the amount of KV state already written to host memory. A nominally safe preemption can become unsafe when several restores compete for the same link. Report raw throughput beside effective throughput, P50 and P99 first-token latency, buffer-underflow time, preemption count, and bytes transferred per switch. Replay bursty traces as well as steady Poisson traffic. Buffer opportunities are largest when arrivals bunch together, so a smooth benchmark can hide both the benefit and the failure mode. Finally, test overload. Raise the sum of requested consumption rates past measured capacity and verify that admission closes cleanly instead of producing an I/O storm. LLM token buffer occupancy scheduling works when unread tokens buy more time than the context switch consumes. Buffer size supplies the numerator, consumption rate turns it into time, and the memory path decides whether that time is usable. --- This representation contains public Fanout content only. Protected Pro lessons, account data, billing, checkout, and pricing are not included. Browse the public content map: https://fanout.sh/sitemap.md