What is KV cache in an LLM?

KV cache (Key–Value cache) is temporary inference-time memory that stores the key (K) and value (V) tensors computed for previously processed tokens in a Transformer's attention layers.

In an autoregressive LLM, each new output token must attend to earlier tokens. Instead of recalculating the earlier keys and values on every step, the model can reuse them from the cache.

What exactly is stored?

In self-attention, a token produces query (Q), key (K), and value (V) representations. The KV cache retains K and V for past tokens, separately for each relevant attention layer. The current token's query can attend to those cached representations.

A cache is not a copy of the original text, the model's trained weights, or a permanent memory of a conversation.

Simple example

Suppose a prompt has three tokens, and the model is generating a fourth:

Prompt:       [A] [B] [C]
KV cache:      K,V K,V K,V    ← created during prefill

Next token:               [D]
Attention:    cached A,B,C + current D
New cache:     K,V K,V K,V K,V

The model reuses the stored attention information from A, B, and C while processing D, and adds D's key/value tensors for later steps.

When is the cache created?

1. Prefill: The model processes the prompt and stores its K/V tensors.
2. Decode: Newly processed tokens contribute additional K/V entries.
3. Request completion: Request-specific cache memory can be released or reused by the serving system.

Some serving systems also support prefix caching, letting requests reuse the computed K/V state of an identical shared prompt prefix when safe and supported.

Why is KV cache important?

• Less repeated computation: Earlier K/V projections do not need to be recomputed for each new token.
• Faster autoregressive decoding: Token generation becomes more efficient.
• Memory cost: Storing K/V tensors for many layers, tokens, and concurrent requests can consume substantial accelerator memory.

For a standard full-attention cache, memory generally grows with the number of cached tokens. The exact amount depends on model layers, KV heads, head dimension, numerical precision, and the number of active sequences. Architectures with grouped-query attention (GQA) or multi-query attention (MQA) can use fewer KV heads; sliding-window caches may retain only part of the context.

What it does *not* do

KV caching does not change what the model has learned, eliminate all attention computation, or make unlimited context free. It trades additional memory for reduced repeated work during inference.

For related topics, see “How does LLM inference work?”, “How does KV caching speed up LLM inference?”, and “Why does KV cache consume significant memory?”

In short: the KV cache is a reusable, per-layer store of past attention keys and values that makes token-by-token LLM inference more efficient.

References: Hugging Face: How caching works and Hugging Face: Cache strategies.