Decoder KV-cache memory

Estimates the raw decoder self-attention key-and-value cache bytes for a homogeneous transformer configuration.

K V cache memory equals two times layer count times K V head count times head dimension times cached tokens times active sequences times bytes per element.

Count every cached key and value element across layers, heads, tokens, and active sequences, then convert elements to bytes.

Multiplication is the main operation because each independent tensor axis expands the number of stored elements.

This is a capacity estimate. Allocator blocks, fragmentation, metadata, cross-attention, prefixes, quantization, and runtime layout can change observed memory.