KV cache formula for LLM inference memory

An LLM can fit on a GPU and still run out of memory when real traffic arrives.

The missing term is often the KV cache. Model weights are mostly static, but the cache grows with every active token and every concurrent sequence.

The useful calculation is small enough to do before choosing hardware.

The KV cache formula

For a standard transformer attention layer, estimate cache bytes as:

2 × layers × KV heads × head dimension × cached tokens × active sequences × bytes per element

The leading 2 accounts for one key tensor and one value tensor.

In symbols:

KV bytes = 2 × L × Hkv × Dhead × T × B × E

Each factor has a concrete meaning:

  • L is the number of cache-producing attention layers
  • Hkv is the number of key-value heads
  • Dhead is the dimension of each head
  • T is the cached token count per sequence
  • B is the number of active sequences
  • E is the number of bytes used by each cache element

Use 2 bytes for FP16 or BF16, 1 byte for an 8-bit cache, and 4 bytes for FP32.

The formula describes the logical tensor payload. A serving engine can require extra memory for allocation metadata, temporary workspaces, CUDA graphs, kernels, and fragmentation.

Query heads and KV heads are not always equal

Multi-head attention gives every query head its own key and value head. In that case, Hkv equals the number of attention heads.

Grouped-query attention lets several query heads share one KV head. If 32 query heads share eight KV heads, the GQA ratio is 4:1.

The cache formula uses eight, not 32.

That one field can change the estimate by a large factor. Copying the visible attention-head count from a model card is not enough.

Multi-query attention is the extreme case: all query heads share a single KV head.

Architectures such as Multi-Head Latent Attention compress the cache differently. Recurrent mechanisms such as KDA keep a fixed-size state instead of a token-level KV entry for every layer.

For those models, use the architecture’s documented cache shape rather than forcing it into the ordinary formula.

Work the units from one token upward

First calculate bytes per token for one sequence:

bytes per token = 2 × L × Hkv × Dhead × E

Then multiply by cached tokens and active sequences.

This order makes the result easier to audit. It also gives a reusable per-token number for admission control.

Consider an illustrative 32-layer GQA layout:

  • 32 query heads
  • Eight KV heads
  • Head dimension 128
  • FP16 cache

The per-token cache is:

2 × 32 × 8 × 128 × 2 = 131,072 bytes

That is exactly 128 KiB for one token in one sequence.

At 16,384 cached tokens and eight active sequences:

128 KiB × 16,384 × 8 = 16 GiB

The cache alone consumes 16 GiB before model weights, activations, runtime workspaces, or safety margin.

MHA versus GQA changes the capacity result

Keep the same 32-layer example, but replace eight KV heads with 32.

The per-token cache becomes 512 KiB. At the same 16,384-token context and batch of eight, the total becomes 64 GiB.

Only the attention layout changed. The workload did not.

On an 80 GiB accelerator, the GQA cache leaves 64 GiB for everything else. The MHA cache leaves 16 GiB.

That does not mean the GQA model fits. Weight memory may already consume most of the device.

It means the cache budget is now visible instead of being discovered through an out-of-memory error.

A larger model can still have a compact cache

Parameter count does not appear in the KV formula directly.

Cache size depends on layers, KV heads, head dimension, token count, batch, and precision.

An illustrative 80-layer layout with eight KV heads and a head dimension of 128 uses 320 KiB per token in FP16.

At 16,384 tokens and eight active sequences, that is 40 GiB.

Two models with similar parameter counts can therefore need different cache budgets. A smaller model with full multi-head attention may use more KV memory than a larger GQA model at the same workload.

Architecture fields matter more than the label on the model.

Context and concurrency multiply

Cached tokens and active sequences are independent multipliers.

Doubling context from 16K to 32K doubles the cache.

Doubling active sequences from eight to 16 also doubles it.

Doubling both makes the cache four times larger.

This is why a model can pass a single-request test and fail under modest concurrency. The weight footprint stayed fixed while the dynamic cache budget expanded.

Production planning should use the distribution of prompt and output lengths, not only the maximum context advertised by the model.

Most requests may be short, but a few long sequences can occupy enough cache blocks to delay or reject new work.

Weight memory is a separate line item

A first-pass weight estimate is:

weight bytes = parameter count × bytes per stored parameter

An FP16 parameter uses roughly two bytes. An 8-bit parameter uses roughly one. A 4-bit weight starts near half a byte before scales, metadata, padding, and any higher-precision weights.

For a mixture-of-experts model, the active parameter count describes compute per token. The total stored expert weights still matter for deployment memory across the cluster.

Do not add active parameters to the KV formula. Keep static weights, dynamic cache, temporary activations, and runtime overhead as separate budget lines.

That separation makes quantization decisions easier to reason about.

Weight quantization reduces the static term. KV-cache quantization reduces the dynamic per-token term. They solve different capacity problems.

PagedAttention improves physical use, not the formula

The PagedAttention paper starts from a serving problem: request caches grow and shrink, and their final lengths are not known in advance.

Preallocating one contiguous region per request creates internal and external fragmentation. It can also prevent related sequences from sharing prefix blocks.

PagedAttention divides the KV cache into fixed-size blocks and allocates them on demand, much like virtual-memory pages.

That does not remove the keys and values required by the model.

It reduces wasted physical memory around the logical cache and enables block-level sharing.

The formula remains the right tensor estimate. The serving engine determines how closely physical allocation approaches it.

vLLM’s prefix-cache design shows how token blocks and their preceding prefixes form reusable cache identities.

Fanout’s PagedAttention Daily note connects the paper’s memory layout to batching and throughput.

Prefix caching changes who pays for prefill

When requests share an identical prefix, a serving engine may reuse the prefix’s KV blocks instead of recomputing them.

This can reduce prefill work and physical duplication. It does not make two different prefixes share a cache merely because their text is similar.

Cache identity normally depends on exact token content, model configuration, and other request inputs that affect the computed state.

Hybrid architectures add more state to validate. The Kimi K3 architecture combines a growing MLA cache with fixed-size KDA recurrent checkpoints.

A reusable prefix is valid only when both mechanisms represent the same boundary.

Turn the estimate into an admission rule

Capacity planning becomes useful when it changes a scheduling decision.

Start with total device memory. Subtract model weights, reserved runtime memory, and a measured safety margin.

The remainder is the cache budget.

Divide that budget by bytes per token to estimate how many cached tokens can be resident across all active sequences.

Then test the result against realistic mixes:

  • Short prompts with long generation
  • Long prompts with short generation
  • Shared prefixes
  • Bursty concurrency
  • Mixed cache precision
  • Preemption or swap behavior

The result is not a promise of throughput. It is a memory boundary the scheduler can enforce before the device fails.

Use the calculator, then verify on the real engine

Fanout’s Inference Memory and KV Cache Lab implements the formula with visible assumptions.

Switch between GQA and MHA, change context, batch, precision, and GPU capacity, and watch the budget move.

The Lab is deterministic and browser-local. It estimates KV tensors, not total deployment memory.

After the estimate, profile the actual serving engine. Measure allocated memory, reserved memory, cache-block utilization, prefix-hit rate, preemption, and out-of-memory behavior.

The formula should get you close enough to ask the right systems questions. The runtime tells you how much overhead your stack adds.

That is the core habit of inference engineering: make the memory model explicit before traffic makes it visible for you.