---
title: "Orca: A Distributed Serving System for Transformer-Based Generative Models"
description: "Orca moves LLM serving from fixed request batches to iteration-level scheduling. Read it to see how a scheduler can reconsider the active batch after every generated token, and how selective batching preserves efficient shared GPU work even when requests have different sequence lengths and lifetimes."
canonical_url: "https://fanout.sh/daily/2026-08-04-orca-continuous-batching"
md_url: "https://fanout.sh/daily/2026-08-04-orca-continuous-batching.md"
last_updated: "2026-08-04"
access: "public"
---

# Orca: A Distributed Serving System for Transformer-Based Generative Models

Orca moves LLM serving from fixed request batches to iteration-level scheduling. Read it to see how a scheduler can reconsider the active batch after every generated token, and how selective batching preserves efficient shared GPU work even when requests have different sequence lengths and lifetimes.

## Paper details

- Authors: Yu et al.

- Venue: OSDI 2022

- Track: Inference

- Difficulty: Easy

- Reading time: 45 min

- Original paper: https://www.usenix.org/conference/osdi22/presentation/yu

## What you will learn

- Why fixed request batches make short generations wait behind long ones and keep new arrivals outside the running work.

- How iteration-level scheduling lets finished requests leave and waiting requests join between autoregressive model steps.

- How selective batching flattens compatible non-attention operations while keeping Attention and KV-cache state separate for each request.

## Continue exploring

[Explore Continuous Batching](https://fanout.sh/blog/continuous-batching-llm-inference): Continue into Fanout's continuous-batching guide for iteration boundaries, changing batch membership, token budgets, KV-cache pressure, and modern serving policy.

---
This representation contains public Fanout content only. Protected Pro lessons, account data, billing, checkout, and pricing are not included.

Browse the public content map: https://fanout.sh/sitemap.md