Continuous batching for LLM inference

Static batching is a poor fit for generated text.

Put several requests in one batch and they will rarely finish together. One answer stops after 18 tokens, another after 300, and a third may hit its maximum length.

If the batch cannot change until every request finishes, completed slots sit idle and new work waits outside.

Continuous batching fixes that lifecycle mismatch by scheduling at model-iteration boundaries. The foundational design appears as iteration-level scheduling in the Orca paper.

Static batches assume a shared finish line

Traditional dynamic batching collects requests for a short window, builds a batch, executes it, then returns the results.

That works well when one model invocation completes one request.

Autoregressive generation is different. A request invokes the model repeatedly, usually producing one new token per decoding iteration.

A batch is therefore not one execution unit. It is a temporary group of sequences at different positions in longer token-by-token lifecycles.

Holding that group fixed wastes a slot whenever a sequence finishes before its neighbors.

It also creates head-of-line blocking. A newly arrived request cannot use the finished slot until the longest sequence in the old batch stops.

Continuous batching reschedules between iterations

An iteration-level scheduler runs one model step for the selected sequences, collects the results, and makes another scheduling decision.

At that boundary:

• Finished sequences can leave.
• Canceled sequences can release their cache.
• Waiting requests can join.
• Active sequences can remain for another decode step.
• Prefill work can be admitted under the token budget.

The batch is continuous because membership changes while the server stays busy. It is not a promise that every arriving request starts immediately.

Admission still depends on available token budget, KV-cache capacity, priorities, and the engine’s scheduling policy.

One iteration may contain uneven work

For active decode sequences, an iteration commonly advances one token per sequence.

New prompts are larger scheduling objects. A prompt might contribute hundreds or thousands of tokens if admitted as a full prefill.

Modern engines can use chunked prefill so a long prompt contributes only part of its tokens to one iteration.

This means “batch size” can be ambiguous.

The number of sequences matters for cache residency and request concurrency. The number of scheduled tokens matters for the work sent through the model in that iteration.

Capacity dashboards should expose both.

The scheduler spends a token budget

Think of each iteration as having a maximum number of tokens it can schedule.

Decode sequences usually request one token each. Prefills request the prompt tokens not yet processed, or a chunk that fits the remaining budget.

The policy decides which requests spend the budget first.

Current vLLM optimization guidance describes a decode-first policy with chunked prefill in V1.

Pending decode tokens are scheduled first. Remaining budget goes to prefill work, and an oversized prefill is split.

That policy protects streaming smoothness, but a different workload or engine may make another tradeoff.

Batch membership and memory allocation are coupled

Admitting a sequence means more than finding arithmetic work for the GPU. The server must also keep that sequence’s KV cache resident.

As generation continues, each active sequence asks for more cache positions.

A scheduler can have room in the iteration’s token budget but no free KV blocks. It can also have free cache capacity while choosing a smaller batch for latency reasons.

This is why continuous batching and PagedAttention are often discussed together.

PagedAttention makes the dynamic cache pool easier to allocate. Continuous batching decides which dynamic set of sequences gets to use it now.

They solve different halves of the same serving loop.

Follow one sequence through the loop

A request arrives with a 600-token prompt.

The scheduler places it in the waiting queue. When token and cache budgets allow, the prompt enters prefill, perhaps as several chunks.

After the first generated token, the request becomes an active decode sequence.

Each later iteration may schedule one new token for it alongside tokens from many other sequences.

When it emits a stop token, reaches a length limit, or is canceled, it leaves before the next iteration. Its KV blocks return to the free pool.

A waiting request may take that capacity immediately. The server does not wait for the rest of the previous batch to finish.

That lifecycle is the practical meaning of continuous batching.

Iteration boundaries create useful control points

Fine-grained scheduling gives the engine more chances to react.

It can favor an older request, reserve capacity for a priority class, stop canceled work, or prevent one long prompt from monopolizing the device.

It can also make poor decisions more frequently.

Complex policies add scheduler overhead, fairness questions, and harder performance debugging. A priority queue can starve low-priority work if the admission rules lack aging or quotas.

Iteration-level scheduling is a mechanism. Fairness and service objectives remain policy choices.

The original Orca design also introduced selective batching because not every transformer operation handles varying sequence lengths in the same way.

Preemption is the pressure-release valve

An active request may need another KV block when the cache pool is full.

The engine then needs a policy: reject new work, pause a sequence, move state elsewhere, or discard state and recompute it later.

Current vLLM V1 documentation describes recomputation as its default preemption mode. Repeated preemption is a warning that the active set exceeds comfortable cache capacity.

Preemption preserves liveness, but it is not free.

Recomputation repeats earlier model work. Swapping moves a large cache across a slower link. Either path can create latency spikes.

Monitor preemption count beside cache utilization and waiting requests. A server can look highly utilized while spending meaningful work recovering from bad admission.

Chunked prefill protects live decodes

Without chunking, one long prompt can become a large iteration and delay every active stream sharing the device.

Chunked prefill gives the scheduler more interruption points. It can place a bounded amount of prompt work beside decode tokens, then reconsider the mix.

A smaller chunk or token budget often helps inter-token latency. A larger budget can complete prompts sooner and improve TTFT.

The detailed prefill versus decode guide explains why those metrics can move in opposite directions.

Continuous batching supplies changing membership. Chunking supplies smaller prompt units. The scheduling policy decides how the two are combined.

Throughput claims need a latency boundary

The Orca evaluation reported a large throughput improvement over its FasterTransformer baseline on GPT-3 175B.

The later vLLM paper compared its paged memory system against Orca-style scheduling and other baselines.

Those results established the importance of iteration-level scheduling and cache management. They should not be copied into a capacity plan as timeless constants.

Hardware, kernels, model shape, prompt lengths, output lengths, and latency targets all change the realized gain.

The responsible comparison holds a latency objective and workload distribution constant, then measures how much request or token throughput the engine sustains.

Watch percentiles, not a single average

Continuous batching improves device use, but aggressive admission can still harm user experience.

Track:

• Queue time before prefill
• TTFT by prompt-length bucket
• Inter-token latency by active-sequence count
• End-to-end latency by output-length bucket
• Scheduled tokens per iteration
• Running and waiting sequence counts
• KV-cache utilization and preemptions
• Output tokens per second

Percentiles matter because one large prefill or cache-pressure event can create a tail spike that the mean barely shows.

The Fanout Scale Lab can turn an estimated workload into an explicit capacity boundary before load tests refine it.

Continuous batching is not request concurrency

An API can accept many concurrent requests while running them through small fixed batches.

Conversely, an engine can use continuous batching but enforce a low concurrency cap to protect latency or memory.

Concurrency describes how many requests are in flight. Continuous batching describes when the engine can change the set executing together.

Do not infer one from the other.

Also distinguish server-side batching from a client sending several prompts in one API call. Client batching fixes the request group at the interface; continuous batching is an internal scheduling behavior.

A practical tuning order

First, replay a representative mix of prompt and output lengths. A uniform synthetic prompt misses the lifecycle problem continuous batching is meant to solve.

Second, set explicit TTFT and inter-token latency targets. Throughput without those boundaries rewards an engine for admitting too much work.

Third, vary the maximum scheduled tokens and sequence count. Watch when throughput flattens while tail latency or preemption climbs.

Fourth, test a burst of long prompts during active decoding. That exposes whether chunked prefill protects existing streams.

Finally, test cancellation and early stopping. Capacity should return promptly when a user disappears or a sequence finishes.

The systems view

Continuous batching is a scheduler, not a magic throughput flag.

It works because autoregressive requests expose a natural boundary after every model iteration. At that boundary, the engine can replace finished work with useful work.

The full loop includes more than scheduling:

1. Admit prompt tokens.
2. Allocate KV blocks.
3. Execute prefill or decode.
4. Stream completed tokens.
5. Retire finished sequences.
6. Reclaim cache and reschedule.

Study those transitions together. A fast scheduler cannot compensate for a fragmented cache, and an efficient cache cannot choose a fair batch.

Fanout’s inference engineering roadmap treats that loop as one system: model execution, KV memory, latency objectives, and admission control belong on the same diagram.