KV-cache memory formula for LLM inference

A model can fit in GPU memory and still run out of room when real requests arrive. The missing object is often the KV cache: a growing record of attention keys and values kept so decoding does not recompute the entire prompt for every new token.

The useful question is not only how many parameters the model has. It is how many bytes one live sequence adds, how that cost scales with concurrent requests, and which architectural choices change the number of stored key-value heads.

This guide derives a raw-cache estimate from tensor shapes, tests its invariants, and then marks the production details the simple formula deliberately leaves outside its boundary.

The cache trades memory for repeated compute

During autoregressive decoding, each new token attends to earlier positions. The key and value projections for those earlier positions do not change, so an inference system can store them once and reuse them at later decoding steps.

Without that reuse, the system would repeatedly project the same prefix tokens. The cache removes that repeated work, but it makes live sequence length and concurrency first-class memory variables.

The original Transformer paper defines the key and value roles inside attention. The memory expression below is a shape-derived systems estimate, not an equation quoted from that paper.

Derive the formula one axis at a time

For one layer, one request, and one cached token, the key tensor stores n kv heads times d head elements. The value tensor has the same shape. That creates the leading factor of two.

Repeat that pair across L layers, T cached tokens, and B concurrent sequences. Finally multiply the number of elements by b bytes per stored element.

The resulting idealized formula is M KV equals 2 times L layers times n kv times d head times T times B times b. Every factor has a visible physical interpretation; no factor should be added merely because it appears in another model's calculator.

Unit checking is decisive. Layers, heads, tokens, and requests are counts. Head dimension is elements per head. Bytes per element converts the final element count into bytes.

Use key-value heads, not query heads

In multi-head attention, every query head has its own key and value head. Multi-query attention shares one key and one value head across all query heads. Grouped-query attention keeps an intermediate number of key-value heads.

The Grouped-Query Attention paper describes GQA as using fewer key-value heads than query heads, positioned between multi-head and multi-query attention. That is why n kv, not the total query-head count, belongs in this cache formula.

Sharing changes cache capacity without changing the number of query heads that produce attention outputs. It can also affect quality, kernel layout, and throughput, so it should not be read as a free memory switch.

A small example exposes the scale

Take 32 layers, 8 key-value heads, head dimension 128, 8,192 cached tokens, batch size 2, and two bytes per element. One token for one request costs 131,072 bytes across all layers.

Multiplying by 8,192 tokens and two requests gives 2,147,483,648 bytes, or exactly 2 GiB under binary units. That is raw K and V storage before allocator blocks, metadata, temporary workspace, or model weights.

The calculation also reveals a reusable rate: 128 KiB per cached token per request for this architecture and precision. Once that rate is known, sequence growth becomes easy to estimate.

Test the linear invariants

Hold everything else fixed and double sequence length: raw cache bytes double. Double concurrent requests: bytes double. Move from two-byte to one-byte cache elements: bytes halve.

Doubling one factor while halving another leaves the estimate unchanged. The workbench uses that conservation test because it checks the formula's multiplicative structure rather than memorizing a particular answer.

At the extremes, an empty cache uses zero bytes in the idealized formula. Multi-query attention minimizes the key-value-head factor at one. Long contexts and large live batches can dominate even when model weights remain fixed.

Explode the cache into its factors

Use the workbench below to switch among MHA, GQA, and MQA, then vary token count, batch size, and bytes per element. Read the factor stack before the final capacity number.

The 24 GiB figure is only a reference budget for comparison, not a claim that 24 GiB is available to the cache. Weights, activations, runtime buffers, fragmentation, and the serving engine all consume memory too.

Open the KV-cache formula case for a symbol ledger and worked example, then use the interaction to test which factor a proposed optimization actually changes.

Raw bytes are not reserved bytes

A production server allocates cache storage in physical layouts. Requests arrive and finish at different times, sequence lengths vary, and blocks may be partially filled. Reserved memory can therefore exceed the logical elements in the formula.

The PagedAttention paper treats KV-cache memory as non-contiguous blocks inspired by virtual-memory paging. Its central concern is allocation and sharing under dynamic serving workloads, not a change to the semantic meaning of keys and values.

Prefix sharing can make several requests reference common cached blocks. Tensor parallelism can shard cache tensors across devices. Offloading moves storage to a different tier. These choices change placement or duplication and must be modeled separately.

State the assumptions before trusting the answer

The workbench assumes a decoder-only stack whose counted layers use the same key-value-head count and head dimension. It assumes each live request has the displayed cached length and stores both K and V at one uniform precision.

The formula excludes model parameters, activations outside the persistent cache, allocator metadata, padding, block slack, kernels' temporary workspace, cross-attention caches, and host-side copies.

Some architectures compress, evict, quantize, window, or otherwise transform the cache. In those systems, replace the relevant factor with the actual stored shape and add any new state rather than forcing the baseline formula to fit.

Ask a mechanism-level question

Suppose context length doubles while the cache moves from two-byte elements to one-byte elements. With all other factors fixed, what happens to raw KV-cache memory.

The answer is that it stays the same: the factor of two from tokens is cancelled by the factor of one-half from element width. This remains a raw-storage statement, not a promise about end-to-end GPU usage.

Frequently asked questions

Does batch mean the configured maximum batch

Not necessarily. In this estimate B is the number of simultaneously live sequence caches represented by the calculation. Continuous batching changes that number over time, so capacity planning needs a workload distribution or a conservative operating point.

Why is prompt length included after prefill

Because later decoded tokens still attend to the prompt unless the architecture uses a restricted window or eviction rule. Prefill creates cache entries for the prompt; decode appends entries for generated tokens.

Does cache quantization always halve total memory

It halves the raw element payload only when element width halves and stored shapes stay fixed. Scales, zero points, alignment, metadata, unsupported layers, and allocator granularity can reduce the realized saving.

Continue from capacity to performance

Practice the full decoding routine in the Math Decoder curriculum, then continue to the Roofline guide to ask whether moving those bytes also limits attainable compute throughput.

The durable habit is to write the stored tensor shape, multiply only the axes that physically exist, preserve units, and label the answer as logical payload or actual reserved memory.