Continuous Batching

Requests arrive when their callers send them and finish when their answers are done, and neither moment lines up with anything the server happens to be doing. An engine that remakes its decision on every forward pass can take them as they come: a request joins the set being worked on at a step boundary, and it leaves on the…

In this lesson

• Joining and leaving the running set
• Thirty-two requests at once
• TTFT, the gaps, and the denominator
• Orca, and two names for one decision
• What this page has not bounded

Continue learning

Pro unlocks the complete lesson, video, practice, and code where available.