How do you prevent cache stampedes and concurrent duplicate LLM requests?

A cache stampede (thundering herd) happens when many requests simultaneously miss an expired or absent entry and each independently triggers expensive LLM generation. The standard defense is request coalescing (singleflight): for an eligible exact cache key, allow one request to refill and let others await its result. Combine this with bounded waiting, safe fallback, rate limits, and carefully planned cache expiry.

1. Why it happens

• A popular answer expires or an application starts with an empty cache.
• Fifty users ask the same FAQ within a short interval.
• Without coordination, all fifty observe a miss, then launch fifty equivalent LLM calls. This can saturate inference concurrency, increase queue latency and multiply token costs.
• A semantic-cache similarity match is not by itself a safe coalescing key: different questions with superficially similar embeddings may need different answers.

2. Beginner-friendly numerical example

Suppose 100 near-simultaneous, identical and authorized requests each need an LLM call costing $0.02 and taking 2 seconds.

• Without coalescing: 100 calls, $2.00 in model charges, and 100 expensive generations competing for capacity.
• With correct singleflight: 1 underlying call, $0.02 in model charges, and up to 99 waiters reuse its verified result, a 99% reduction in duplicate generation calls.
• The 99 waiters still incur waiting time and cache/coordination overhead. Overall latency and total costs are not automatically reduced by 99%. These numbers are hypothetical.

3. End-to-end implementation

1. Build an exact safe key from canonicalized request inputs plus tenant/authorization scope, model, system prompt, retrieval/document versions, relevant tool context and generation settings. Never coalesce calls across incompatible contexts.
2. Check the cache. On a hit, enforce freshness and permissions before returning the stored answer.
3. On a miss, elect one owner of the in-flight generation for this key. Other callers join its pending result rather than issuing duplicate model requests.
4. Publish carefully: verify the completed response, check that source versions and permissions still match, and then populate the cache. Wake waiters.
5. Handle failure: impose deadlines, limit waiter count, retry with backoff and jitter if appropriate, and release the ownership marker on success or error. A failed first request should not trap all waiters forever.

4. One process versus many replicas

• Single process: use an in-memory per-key future, promise or lock (often called *singleflight*). This deduplicates work only within that process.
• Multiple replicas: consider a shared short-lived lease or distributed lock (e.g. Redis), plus a shared result cache and notifications or bounded polling. A local lock in each replica does not prevent cross-replica duplicate LLM calls.
• Safe lease handling: use an owner identifier, expiry, and conditional release; stale owners must not overwrite a newer result. Fencing/version checks matter when correctness is important. Do not assume lock TTL means the owner remains valid until its work ends.

5. Other complementary protections

• TTL jitter: stagger expiration across different keys so thousands of entries do not all expire together. It does not solve concurrent misses for one hot key.
• Stale-while-revalidate: briefly serve an older response while one worker refreshes only if the answer is still authorized and safe to serve stale. Never use this to bypass urgent policy or permission invalidation.
• Proactive refreshing: refresh popular, approved entries before TTL expiry while respecting rate limits.
• Admission controls: cap concurrent LLM calls and waiting requests; use backpressure rather than allowing unbounded memory/queue growth.
• Streaming: sharing one live output stream among callers needs special care for cancellation, backpressure, tool side effects and privacy; do not blindly merge personalized or action-taking requests.

6. Production monitoring

Track cache miss bursts, duplicate-generation suppression ratio, singleflight waiters, lock contention and timeouts, generation error rates, queue depth, p95/p99 TTFT and cost per successful response. Test simultaneous requests, leader crashes, lease expiry, source-version changes and tenant boundaries.

Interview-ready answer: “A stampede occurs when concurrent misses launch duplicate LLM work. I'd use per-key singleflight for identical authorized inputs, with a shared lease across replicas where needed. TTL jitter spreads expiry across keys; safe stale-while-revalidate and proactive refresh reduce misses. I'd bound waiting and handle owner failures while preventing obsolete results or cross-tenant data reuse.”

Research: Redis cache layer architecture guide; Redis distributed locks; AWS CloudFront stale-while-revalidate.