Chunked Prefill & Choosing the Chunk Size

A prompt is computed before its first token comes back, and so far that computation has been indivisible. A request either entered a pass with room for the whole prompt, or it waited for a pass that had room. The scheduler budgets lesson fixed the size of that room, and named the one place where a request's length and a…

In this lesson

• A prompt against a budget
• The clamp
• What splitting changes
• The default and the platform
• Back to the startup error

Continue learning

Pro unlocks the complete lesson, video, practice, and code where available.