KV Cache

During generation, every new token would naively recompute attention keys and values for the whole history. The KV cache stores them once and reuses them, turning quadratic regeneration into linear appends — the single most important inference optimization, and the reason long chats eat GPU memory.