Orca: A Distributed Serving System for…

Orca moves LLM serving from fixed request batches to iteration-level scheduling. Read it to see how a scheduler can reconsider the active batch after every generated token, and how selective batching preserves efficient shared GPU work even when requests have different sequence lengths and lifetimes.

Reading focus: Why fixed request batches make short generations wait behind long ones and keep new arrivals outside the running work. How iteration-level scheduling lets finished requests leave and waiting requests join between autoregressive model steps. How selective batching flattens compatible non-attention operations while keeping Attention and KV-cache state separate for each request.

OSDI 2022. Yu et al.. 45 min read, easy difficulty.