How does prefix caching differ from prompt caching, and when should each be used?

Prefix caching and prompt caching are often two names for the same underlying optimization: reusing previously computed attention KV states for an identical prompt prefix. They should not be described as fundamentally different algorithms. In practice, prefix caching often refers to the low-level mechanism in a self-hosted serving engine, while prompt caching is the user-facing feature exposed by a model API. The real trade-offs are control, pricing, eviction, routing, supported prefixes, and security.

1. Understand the common mechanism

• A Transformer computes attention key/value (KV) states while processing prompt tokens (prefill).
• If the next request has the same initial token sequence under a compatible execution context, it may reuse those states rather than recomputing them.
• A cache hit saves eligible prefill work; it does not cache the final generated answer or automatically make output decoding faster.
• The same prefix principle powers vLLM Automatic Prefix Caching (APC) and provider API features called prompt caching. <cite>vLLM design; OpenAI and Anthropic documentation</cite>

2. What changes between deployments?

Self-hosted prefix caching (for example, vLLM):

• Your serving engine manages KV blocks, hashes, reference counts and eviction as part of GPU memory management.
• You may control enablement, memory budgets, scheduling, security boundaries and cache-aware routing.
• There is normally no separate provider cached-token tariff, but cached blocks consume your GPU capacity. Any cost reduction comes from improved efficiency and throughput.

Managed API prompt caching (for example, OpenAI or Claude):

• The provider manages storage, eviction, routing, and cache availability.
• The API may offer automatic prefix matching, explicit breakpoints, TTL/retention controls, and token-usage accounting—depending on the model and provider.
• Cached input tokens may have different billing rates from uncached tokens; cache writes may also carry an explicit price.
• A provider cache may have minimum qualifying prompt lengths and other restrictions. Always inspect the current model's docs.

3. Beginner-friendly numerical example

Suppose your application asks 10 questions, each using an identical 6,000-token document followed by a distinct 200-token question.

Without reusable caching:

• 10 × (6,000 + 200) = 62,000 prompt tokens processed.

With a perfect shared-prefix hit on requests 2–10:

• First request computes 6,200 tokens.
• Nine later requests only require 200 new question tokens each (plus attention to cached states).
• Newly processed tokens ≈ 6,200 + 9×200 = 8,000.
• Approximate eligible prompt-token computation avoided = 54,000 / 62,000 = 87.1%.

This is an idealized computation comparison, not an API-billing or wall-clock-speed guarantee. Cache block sizes, minimum cacheable length, cache expiry, provider-specific accounting, and the actual attention work of new tokens affect results.

4. When should I choose which?

• Running your own open-weight model? Prefer serving-engine prefix caching with vLLM or another engine when requests share long identical beginnings and you can manage GPU KV memory.
• Calling a hosted model API? Use its documented prompt caching controls, if available. Keep system instructions, tool definitions and reference material stable and at the front; place changing user content later.
• High-reuse chatbot/agent workloads? Both approaches may save substantial prefill compute when conversation history or large tool definitions repeat.
• Mostly unique or short prompts? Cache hit rates may be low and the benefit small or unavailable. Do not add unnecessary prompt tokens just to force eligibility.
• Sensitive multi-tenant deployments? Respect tenant isolation, cache-salt/routing policies and provider data-retention requirements. Cross-tenant performance leakage or incorrect reuse is a security concern.

5. Important misconceptions

• Not semantic caching: two differently worded prompts with similar meaning do not normally share their KV cache. Semantic caches typically store/retrieve previously generated results and introduce separate correctness risks.
• Not response caching: prefix reuse still runs the model to generate a fresh output.
• Not a guarantee of a hit: even identical prefixes can miss due to eviction, expiry, different model settings or routing to another cache pool.
• Not necessarily cheaper in every API: compare cache-write charges, read discounts, TTL and traffic reuse patterns for the exact provider/model.

6. Production evaluation checklist

Measure cached input tokens, cache hits, prefill TTFT, end-to-end latency, output throughput and total cost per successful request. For self-hosted serving also watch GPU KV occupancy, evictions and request preemptions. For APIs verify supported breakpoints, matching rules, cache lifetime and retention.

Interview-ready answer: “Both reuse previously computed KV states for matching prompt prefixes. Prefix caching usually names the serving-engine mechanism; prompt caching typically names a provider-managed API feature. The right choice depends on whether we host the model, how often prompts share exact prefixes, cache retention and memory, tenant security, and total latency/cost—not on a fundamentally different attention algorithm.”

References: vLLM Automatic Prefix Caching design; OpenAI API prompt caching; Claude prompt caching.