Why does KV cache consume significant memory?

KV cache consumes significant memory because an autoregressive LLM stores two attention tensors—keys (K) and values (V)—for every retained token across its attention layers. These tensors are kept during inference so subsequent tokens can reuse earlier attention computations instead of recalculating them.

Why does it grow so large?

• Two tensors per token: Both K and V must be retained.
• Many Transformer layers: Each layer maintains its own cache.
• Long prompts and answers: In standard full attention, every additional retained token adds cache entries.
• Concurrent users: Each active request needs cache state; unless a serving system shares identical prefixes, total memory increases with simultaneous sequences.
• Numerical precision: FP16/BF16 tensors use 2 bytes per element; lower-precision KV formats may use less.

How do we estimate KV-cache memory?

For a conventional full-attention decoder with uniform layer dimensions:

KV bytes ≈ 2 × layers × batch_size × cached_tokens
           × KV_heads × head_dimension × bytes_per_element

The leading 2 represents the separate key and value tensors. Use the number of KV heads, which can be smaller than the number of query heads in grouped-query attention (GQA).

Practical example

Consider an illustrative model with:

• 32 attention layers
• 8 KV heads per layer, each of dimension 128
• 8,192 cached tokens in one request
• FP16 KV tensors (2 bytes per element)

Memory = 2 × 32 × 1 × 8,192 × 8 × 128 × 2
       = 1,073,741,824 bytes
       = 1 GiB

That is approximately 1 GiB of raw KV-cache data for one sequence, excluding model weights, temporary tensors, allocator overhead, and any reserved but unused capacity.

Eight independent requests of that length would require roughly 8 GiB for their raw KV data alone if no cache entries are shared.

How can systems reduce the memory cost?

• GQA or MQA: Models designed with fewer KV heads store fewer key/value elements.
• KV-cache quantization: Stores cache tensors with fewer bits, potentially adding accuracy or speed trade-offs.
• Sliding-window attention: For models supporting it, retains only a bounded recent context in applicable layers.
• Paged KV-cache allocation: Manages cache in blocks to reduce wasted reserved memory and fragmentation; it does not remove the underlying need to store K/V data.
• Cache offloading: Moves some cache state to CPU memory, saving GPU capacity but possibly increasing latency.
• Shorter contexts and careful concurrency limits: Reduce the number of live cached token positions.

Important distinction

Model weights are persistent learned parameters; KV cache is temporary request-specific attention state. Even after model-weight quantization, KV cache may remain a major GPU-memory cost during long-context, high-concurrency serving.

See “What is KV cache in an LLM?” and “How does KV caching speed up LLM inference?” for the mechanism.

In short: KV-cache memory grows with retained tokens, KV heads, layers, precision, and concurrent sequences. Caching buys faster decoding at the cost of considerable working memory.

References: Hugging Face: How caching works, Hugging Face: Cache strategies, and vLLM: KV-cache configuration.