Build a KV Cache
Upgrade a generation loop to the cached form every serving system runs, inspect the KV cache tensors and their shapes, and prove the output is unchanged.
Upgrade a generation loop to the cached form every serving system runs, inspect the KV cache tensors and their shapes, and prove the output is unchanged.