RadixAttention, Eviction & Cache Churn

Keeping a block because someone might send its tokens again costs the space that block sits in, and some other request wanted that space. Nothing in this module has charged that cost yet. Both halves of reuse are on the page now, the lookup the automatic prefix caching lesson opened and the traffic the shared prompt reuse lesson shaped for…

In this lesson

• Two structures for one question
• When a block stops being findable
• The operator's version, and the gate in front of it
• Churn

Continue learning

Pro unlocks the complete lesson, video, practice, and code where available.