Large Language Models

Large language models are deep learning models, usually transformer-based, trained to predict tokens from large corpora. After pretraining, they are often instruction-tuned and aligned so they can follow prompts and assist users.

The product team wants a 2× speedup on our Llama 3 70B endpoint without quantization or pruning. How can we accelerate inference without changing the model’s target output distribution, and what Transformer asymmetry makes it possible?

Start with exact speculative decoding, then benchmark it alongside serving optimizations. A 2× speedup is a goal, not a guarantee . An autoregressive 70B model normally generates one token per sequential decode step. At small batch sizes, each step may be limited by repeatedly reading large model weights from GPU memory, while there is spare arithmetic capacity. A small draft model can propose several candidate tokens cheaply, and the 70B target can verify them in parallel during one forward pass. Proper rejection sampling verification preserves the target model’s sampling distribution, rather than accepting lower quality draft outputs. 1. The core Transformer asymmetry Ordinary decode: one new token per target forward pass, with repeated memory movement and relatively little parallel arithmetic per request. Speculative verification: many proposed positions can be evaluated simultaneously by the target Transformer under the causal mask. The target still determines accepted output; it does not blindly trust the draft. Prefill versus decode: prefill has more token parallel work, while small batch decode is often bandwidth bound. This difference makes spare compute available for verification, but it depends on hardware, batch size and context length. 2. Beginner friendly numerical example Suppose a 70B endpoint ordinarily takes 50 ms per generated token for a given request. For 100 tokens, the simplified decode time is 5 seconds , ignoring prefill and network overhead. Now imagine a candidate speculative design where each iteration spends 20 ms drafting and 70 ms verifying , and produces three accepted output tokens on average : Time per emitted token ≈ (20 + 70) / 3 = 30 ms . For 100 tokens, approximate decode time becomes 3 seconds , a 1.67× speedup . A 2× gain would require an effective 25 ms/token under this baseline. If draft acceptance drops or verification becomes expensive, speculative decoding may even slow things down. These are hypothetical performance values , not Llama 3 70B benchmarks. Count any bonus accepted token and sampling overhead in actual measurements. 3. What is genuinely lossless? Exact speculative sampling: a correct rejection sampler produces the same distribution as sampling the target model alone, up to practical floating point differences; it does not guarantee the same sampled token string on every run. Greedy decoding: verified candidates can match the target’s greedy choices, subject to numerical and implementation differences. Not automatically lossless: accepting draft tokens without proper verification, lossy rejection schemes, approximate attention or unvalidated precision changes. KV cache: preserve the target model’s correct causal history and position accounting when accepting or rejecting candidate tokens. 4. How I'd accelerate the 70B endpoint 1. Profile real traffic: separate time to first token (TTFT), inter token latency (ITL/TPOT), output throughput, memory bandwidth, prompt lengths, batch sizes and tail latency. 2. Test exact speculative decoding: choose an efficient draft compatible with the target tokenizer and model; tune the number of proposed tokens and measure acceptance rate and target verification cost. 3. Use kernel and runtime improvements: benchmark FlashAttention/optimized attention kernels, fused operations, CUDA graph capture where supported, suitable layouts and reduced scheduling overhead. Some low level optimizations can introduce numerical differences; verify the output contract. 4. Use KV caching, prompt prefix reuse and efficient paging to avoid redundant work where applicable. Prefix caching is most useful when prompts share a substantial prefix. 5. Tune continuous batching and chunked prefill according to workload. They may increase throughput or improve latency, but are not interchangeable with speculative decoding. 6. Retest against the same precision and accuracy contract at realistic concurrency. A setup that speeds up low concurrency decoding may worsen high batch throughput. 5. Acceptance criteria Compare baseline against candidates on p50/p95 TTFT , ITL , requests/sec, generated tokens/sec, end to end latency, cost/request, peak KV cache memory, draft acceptance and output quality/distribution checks. Record the serving software version, GPU topology and decoding parameters. Select the fastest configuration meeting product SLOs rather than promising exactly 2×. Interview ready answer “Large model decode is sequential across output tokens and often memory bandwidth bound at low batch sizes, whereas verification can evaluate several candidate positions in parallel. I'd try exact speculative decoding with a fast draft model and target rejection sampling, preserving the target distribution. Then I'd profile attention kernels, KV/prefix caching and scheduling. The gain depends on acceptance rate and workload, so I'd benchmark and validate rather than guarantee 2×.” References: Leviathan et al., Fast Inference from Transformers via Speculative Decoding, ICML 2023; vLLM speculative decoding documentation.

What is prefix caching in LLM inference, and how does it reduce redundant computation?

Prefix caching (also called automatic prefix caching or prompt prefix KV reuse) stores the key/value attention states computed for an earlier prompt prefix and reuses them when a later request begins with the same token prefix . This avoids repeating part of the prefill calculation and can reduce time to first token (TTFT) for repeated long prompts. 1. Why do we need it? A standard KV cache stores attention keys and values within one generation request , so decoding doesn't recompute all earlier tokens. Prefix caching goes across requests: if separate requests share the same beginning, their already computed KV blocks can be reused. Common examples: a long shared system prompt; many questions about the same document; follow up turns sharing the same conversation history. The match is on token sequences and compatible execution context , not merely semantically similar wording. 2. Step by step example Imagine an application sends the same 8,000 token instruction and reference document , followed by a different 200 token user question each time. 1. The first request processes 8,200 prompt tokens and creates their attention KV states. 2. Later requests have the same initial 8,000 tokens ; the serving engine finds reusable KV cache blocks. 3. A later request needs new prefill work for roughly 200 question tokens , instead of all 8,200 tokens , assuming full eligible prefix reuse. 4. 8,000 / 8,200 ≈ 97.6% of prompt tokens can avoid redundant KV computation in this idealized case. Important: That is 97.6% fewer newly processed prompt tokens , not a guaranteed 97.6% latency reduction. New tokens still attend to the full cached context; block boundaries, scheduling, memory transfers and output decoding remain. 3. What is actually cached? Each Transformer layer stores key and value tensors for previous token positions. In systems such as vLLM, reusable cache blocks are associated with hashes that account for their own token IDs and the preceding prefix . Additional identity information may be required, such as model/adaptor configuration and multimodal content; cache reuse must respect request compatibility. The blocks consume memory even after an earlier request completes, until they are evicted or reused. 4. When is prefix caching useful? Many users share a long, identical system prompt or reference context. Repeated questions use the same fixed document at the beginning of the prompt. Multi turn chats reproduce a common prior conversation prefix. The workload is prefill heavy , and a substantial share of requests have cache hits. It is less helpful when users have unrelated prompts, shared content appears only after different leading tokens, cached blocks have been evicted, or latency is dominated by generating long outputs . Prefix caching generally does not accelerate decode operations themselves. 5. Security and correctness Do not reuse KV states across incompatible models, LoRA adapters, tokenizers, position encodings or other relevant context. Shared servers need tenant isolation. A cache hit may reveal information about previous requests via timing, even if their content is never returned. vLLM supports cache salts so blocks are reused only among requests allowed to share a trust boundary. Hash design matters; robust collision resistant hashing lowers risks of accidental or unsafe reuse. Verify equivalence with an uncached baseline, within expected floating point differences. 6. Practical evaluation Test repeated and unique prompts separately. Measure: Prefix cache hit rate , matched prefix tokens and saved prefill work. p50/p95 TTFT , prefill throughput and end to end latency. GPU KV cache occupancy, evictions, fragmentation and concurrency. Whether the optimization improves throughput without unacceptable memory pressure or tenant isolation risks. Interview ready answer: “Ordinary KV caching avoids recomputing previous tokens during one request. Prefix caching additionally reuses those KV states across requests with an identical leading token sequence, reducing repeated prefill work and often TTFT. It helps long shared prompts or documents, but not unrelated inputs or output heavy decoding. I'd benchmark cache hit rates and latency while enforcing tenant isolation.” References: vLLM: Automatic Prefix Caching design; vLLM: Automatic Prefix Caching feature documentation.

What is prefix-cache eviction, and how does cache memory pressure affect LLM inference performance?

Prefix cache eviction removes previously stored, reusable KV cache blocks when an LLM serving engine needs their memory for new or active requests. Prefix caching speeds up repeated prompt prefill, but GPU KV memory is finite: the scheduler must balance retaining reusable prefixes against serving current generations. 1. How does eviction work? A completed request can leave behind cached KV blocks available for reuse. They may have zero active references while still containing reusable data. In vLLM's block cache design, when another request needs capacity, a reusable block with reference count zero can be reclaimed. Cached blocks are selected from an LRU style free queue rather than arbitrarily deleting KV state currently required by an executing request. A block still needed by active requests cannot simply be discarded as an ordinary idle cache entry. Removing a cached block's hash association means a future request can no longer reuse that block; if it needs those tokens, the server must recompute their attention states. Some older descriptions additionally explain preferring deeper prefix blocks on recency ties. Follow the running engine version's documented policy rather than assuming every implementation is identical. 2. Beginner friendly numerical example Imagine a simplified pool of 12 equal sized KV blocks : Active requests hold 7 blocks . Completed requests left 5 reusable prefix blocks ; none are currently referenced by an active request. A newly admitted request needs 3 new blocks . If no genuinely unused blocks remain, the engine can evict 3 idle cached blocks to make room while keeping the 7 active blocks and 2 surviving reusable blocks. Suppose the evicted blocks represented 48 common prompt tokens (16 tokens/block). A later request may need to recompute those 48 tokens, rather than getting an immediate cache hit. The numbers are illustrative. Real KV managers may handle allocation, partial blocks and overlapping prefix matches differently. 3. How does this affect latency and throughput? 1. More cache hits: repeated long prompts skip eligible prefill computations, often lowering time to first token (TTFT) . 2. More eviction: cache hit probability falls; previously cached prefix tokens are prefetched and computed again, increasing GPU work and potentially TTFT. 3. High active KV occupancy: many concurrent requests or long contexts can leave too few blocks to admit more requests. Queueing and tail latency rise even if unused accelerator arithmetic capacity remains. 4. Insufficient active capacity: when the serving engine cannot allocate KV blocks required by running sequences, it may use request preemption and recomputation . This is distinct from evicting unreferenced prefix cache entries. 5. Excessive retention: using memory to keep cache entries can be counterproductive if it displaces capacity needed by active requests; a correct allocator can reclaim eligible idle cache blocks on demand. 4. Prefix cache eviction versus request preemption Idle prefix cache eviction: drop a reusable block that no running request needs; future matching prompts become misses. Current execution correctness is preserved. Request preemption: pause or reschedule an active request because the KV budget cannot support it. Depending on the engine, some state may need to be recomputed when resumed. This directly hurts request latency. Do not equate cache eviction counts with active request failures or OOM events. Eviction of idle data can be routine, healthy operation. 5. What should production engineers monitor? Prefix cache queries and hits , and changes in hit rate across repeated prompt workloads. KV cache usage, active and waiting request counts, and cache block residence/idle to eviction times. p50/p95/p99 TTFT , prefill throughput, time between output tokens and overall SLO compliant goodput. Long context traffic, prompt sharing distributions, preemptions and available headroom. For example, vLLM exposes counters such as vllm:prefix cache queries and vllm:prefix cache hits, a usage gauge vllm:kv cache usage perc, and sampled eviction/residence metrics in supported versions. 6. Practical optimization checklist 1. Measure first: distinguish expected cache misses (unique prompts) from lost reuse caused by evictions. 2. Make shared prompts prefix identical whenever appropriate; place stable system instructions and shared context before variable user content. 3. Avoid excessive concurrency or unbounded long contexts when active KV cache pressure causes queueing and preemption. 4. Tune KV memory allocation and scheduler limits under realistic load; reduce competing memory use or scale replicas when justified. 5. Assess locality and tenant boundaries: route recurring prefixes to compatible cache pools only where security and workload isolation permit. 6. Benchmark tradeoffs: compare cache hit rate, memory occupancy, p95 TTFT and completed requests per second, not hit rate alone. Interview ready answer “Prefix cache eviction reclaims KV blocks from completed requests when memory is needed. An LRU like policy can evict zero reference cached blocks without interrupting active requests, but future matching prefixes must be recomputed. At high KV pressure, admission delays or active request preemption may also hurt latency. I'd monitor hits, block residency, KV usage, waiting requests and p95 TTFT, then tune cache capacity and concurrency around real workload reuse.” Sources: vLLM Prefix Caching Design; vLLM Metrics Reference.

How does prefix caching differ from prompt caching, and when should each be used?

Prefix caching and prompt caching are often two names for the same underlying optimization: reusing previously computed attention KV states for an identical prompt prefix. They should not be described as fundamentally different algorithms. In practice, prefix caching often refers to the low level mechanism in a self hosted serving engine, while prompt caching is the user facing feature exposed by a model API. The real trade offs are control, pricing, eviction, routing, supported prefixes, and security. 1. Understand the common mechanism A Transformer computes attention key/value (KV) states while processing prompt tokens (prefill). If the next request has the same initial token sequence under a compatible execution context, it may reuse those states rather than recomputing them. A cache hit saves eligible prefill work; it does not cache the final generated answer or automatically make output decoding faster. The same prefix principle powers vLLM Automatic Prefix Caching (APC) and provider API features called prompt caching . <cite vLLM design; OpenAI and Anthropic documentation</cite 2. What changes between deployments? Self hosted prefix caching (for example, vLLM): Your serving engine manages KV blocks, hashes, reference counts and eviction as part of GPU memory management. You may control enablement, memory budgets, scheduling, security boundaries and cache aware routing. There is normally no separate provider cached token tariff , but cached blocks consume your GPU capacity. Any cost reduction comes from improved efficiency and throughput. Managed API prompt caching (for example, OpenAI or Claude): The provider manages storage, eviction, routing, and cache availability. The API may offer automatic prefix matching, explicit breakpoints, TTL/retention controls, and token usage accounting —depending on the model and provider. Cached input tokens may have different billing rates from uncached tokens; cache writes may also carry an explicit price. A provider cache may have minimum qualifying prompt lengths and other restrictions. Always inspect the current model's docs. 3. Beginner friendly numerical example Suppose your application asks 10 questions , each using an identical 6,000 token document followed by a distinct 200 token question . Without reusable caching: 10 × (6,000 + 200) = 62,000 prompt tokens processed . With a perfect shared prefix hit on requests 2–10: First request computes 6,200 tokens. Nine later requests only require 200 new question tokens each (plus attention to cached states). Newly processed tokens ≈ 6,200 + 9×200 = 8,000 . Approximate eligible prompt token computation avoided = 54,000 / 62,000 = 87.1% . This is an idealized computation comparison, not an API billing or wall clock speed guarantee . Cache block sizes, minimum cacheable length, cache expiry, provider specific accounting, and the actual attention work of new tokens affect results. 4. When should I choose which? Running your own open weight model? Prefer serving engine prefix caching with vLLM or another engine when requests share long identical beginnings and you can manage GPU KV memory. Calling a hosted model API? Use its documented prompt caching controls, if available. Keep system instructions, tool definitions and reference material stable and at the front; place changing user content later. High reuse chatbot/agent workloads? Both approaches may save substantial prefill compute when conversation history or large tool definitions repeat. Mostly unique or short prompts? Cache hit rates may be low and the benefit small or unavailable. Do not add unnecessary prompt tokens just to force eligibility. Sensitive multi tenant deployments? Respect tenant isolation, cache salt/routing policies and provider data retention requirements. Cross tenant performance leakage or incorrect reuse is a security concern. 5. Important misconceptions Not semantic caching: two differently worded prompts with similar meaning do not normally share their KV cache. Semantic caches typically store/retrieve previously generated results and introduce separate correctness risks. Not response caching: prefix reuse still runs the model to generate a fresh output. Not a guarantee of a hit: even identical prefixes can miss due to eviction, expiry, different model settings or routing to another cache pool. Not necessarily cheaper in every API: compare cache write charges, read discounts, TTL and traffic reuse patterns for the exact provider/model. 6. Production evaluation checklist Measure cached input tokens, cache hits, prefill TTFT, end to end latency, output throughput and total cost per successful request . For self hosted serving also watch GPU KV occupancy, evictions and request preemptions. For APIs verify supported breakpoints, matching rules, cache lifetime and retention. Interview ready answer: “Both reuse previously computed KV states for matching prompt prefixes. Prefix caching usually names the serving engine mechanism; prompt caching typically names a provider managed API feature. The right choice depends on whether we host the model, how often prompts share exact prefixes, cache retention and memory, tenant security, and total latency/cost—not on a fundamentally different attention algorithm.” References: vLLM Automatic Prefix Caching design; OpenAI API prompt caching; Claude prompt caching.

What is semantic caching in LLM applications, and how does it differ from KV-cache reuse?

Semantic caching stores a previously validated LLM answer and reuses it when a new question is sufficiently similar in meaning. Unlike KV/prefix caching , which reuses Transformer attention states but still generates a new answer, a semantic cache hit can bypass the LLM response generation entirely. This is an application layer optimization and must be treated as potentially approximate. 1. How semantic caching works 1. An application converts the incoming question into a numerical embedding vector . 2. It searches embeddings of past questions in an indexed cache. 3. If the nearest permitted entry passes a similarity threshold and required metadata filters, it returns the stored, approved response. 4. Otherwise it calls the LLM (and any required retrieval/tools), checks the result, then stores the eligible question answer pair with metadata and an expiry time. 2. Beginner friendly numerical example A support bot previously answered: “How do I reset my password?” Its vetted response has been cached. A new question, “I forgot my password—how can I change it?” , receives a hypothetical cosine similarity 0.96 . The configured threshold is 0.93 , so the cache may return the existing answer if the tenant, language, policy and knowledge version match. Another query, “How do I reset somebody else's password?” , might also score 0.94 despite requiring a different authorization and safety response . Similarity alone is not sufficient: security rules must reject inappropriate reuse. The scores and threshold are illustrative, not universal recommended values . 3. Numerical latency and cost example Assume 1,000 FAQ requests: Without a semantic cache, each requires one LLM call → 1,000 model calls . Suppose 400 requests become correct, eligible cache hits → only 600 model calls . Model call count falls 40% , but total latency and cost savings are not guaranteed to equal 40% : every request may still require embeddings and cache lookup, and entries must be maintained. 4. Semantic caching versus KV/prefix caching What is stored: semantic caching stores a question embedding, completed response , and policy/context metadata; prefix caching stores attention KV tensors for an exact compatible token prefix. Matching: semantic caching uses approximate meaning similarity plus strict filters; prefix caching uses exact eligible token prefix reuse. LLM generation on a hit: semantic caching skips it ; prefix caching still generates the new completion. Correctness: semantic caching can produce false hits or stale answers ; compatible KV reuse preserves the same intended attention computation subject to implementation/numerical conditions. Where it runs: semantic caching usually lives in the app/gateway and a vector capable cache store; KV caching is in the model serving runtime or provider infrastructure. Both can be combined: try an authorized semantic cache first, and on a miss call a model endpoint that supports prefix caching. 5. Where does semantic caching help? Stable, repetitive FAQ responses; product help explanations; fixed policy information that is properly versioned. Many differently worded questions with the same verified answer . Applications with enough repeated volume to pay back embedding and storage overhead. Avoid or heavily restrict it for personalized account answers, medical/legal/financial judgments, live prices, recent events, tool actions, database transactions and any query whose answer depends on unseen user specific state. Do not cache agent tool side effects as though the action had executed again. 6. Production safeguards 1. Apply hard metadata filters before vector matching : tenant/user access, locale, model/prompt version, document version and authorization scope. 2. Use tight similarity thresholds initially, and evaluate false positive hits on a labeled test set. A high cosine score does not prove equivalent intent. 3. Enforce time to live (TTL) and event driven invalidation when source documents, policies or permissions change. 4. Cache only approved, safe and reusable responses , not raw secrets or private answers for cross user use. 5. Monitor hit rate, incorrect hit rate , stale answer incidents, embedding/lookup latency, end to end latency and cost per valid answer. Compare against an uncached baseline. Interview ready answer “Semantic caching returns an existing answer to a semantically similar query, using embeddings, a threshold and strict context filters. KV/prefix caching reuses intermediate attention computations for an identical prefix but still invokes the model. Semantic caching can save more generation work on repeated FAQs, but it introduces false match, stale answer and cross tenant leakage risks, so correctness, authorization and invalidation matter more than maximizing hit rate.” References: Redis semantic caching; AWS ElastiCache semantic caching overview; Azure Managed Redis semantic cache best practices.

How should similarity thresholds be selected and evaluated in LLM semantic caching?

Choose a semantic cache similarity threshold from labeled, production like examples—not intuition. A lower similarity threshold can increase cache hits but also return incorrect answers. A higher threshold is safer but may miss useful paraphrases. The objective is correct, authorized cache reuse at acceptable latency and cost , not maximum hit rate. 1. Understand the score Embedding similarity estimates how close two questions are in vector space; it is not the probability their answers are interchangeable . For cosine similarity where larger means closer, cache a response only if score ≥ threshold and all mandatory context/permission filters pass. Some vector databases instead return cosine distance , where smaller is better; thresholds are reversed. Check your embedding normalization and metric. No universal score such as 0.85 or 0.95 works across all models, datasets and use cases. 2. Beginner friendly labeled example Suppose five new requests are compared to their best permissible cached candidates: Query A: 0.98 , genuinely reusable answer. Query B: 0.95 , genuinely reusable answer. Query C: 0.93 , wrong cached answer despite similar wording. Query D: 0.89 , genuinely reusable answer. Query E: 0.83 , different answer required. At threshold 0.90 , A, B and C are returned: 2 correct hits and 1 incorrect hit . Precision of served hits = 2/3 = 66.7% ; recall of the three reusable candidates = 2/3 = 66.7% . At threshold 0.95 , A and B are returned: 2 correct out of 2 hits (100% precision) , with recall still 2/3 = 66.7% . Query D remains a missed opportunity. All scores and labels are hypothetical; small samples cannot establish real world confidence. 3. How to tune it systematically 1. Build labeled pairs: include exact duplicates, paraphrases, near misses, different intents with similar phrasing, account specific requests and changing facts. Label whether the same full answer is valid , not just whether topics are similar. 2. Separate evaluation: split by conversation/user/source/time where appropriate to prevent duplicate leakage. Keep a final untouched holdout. 3. Sweep thresholds: for example 0.80, 0.85, 0.90, 0.95 and 0.98 only as test candidates . 4. Measure: cache precision = correct served hits / all served hits; recall of reusable answers; false hit rate; correct hit rate across all requests; p50/p95 end to end latency; LLM calls saved; embedding/search cost. 5. Choose based on risk: for ordinary stable FAQs, moderate residual risk may be tolerable with human review. For private, regulated, transactional, rapidly changing or security sensitive answers, disable semantic response reuse or require stronger verification. 6. Validate online: shadow evaluate cache suggestions without serving them initially, audit disagreements, then run a controlled rollout with rollback thresholds and incident monitoring. 4. Guardrails beyond similarity Filter tenant, user authorization, language, tool/state context, document version and policy version before nearest neighbor matching. Use TTL and source change invalidation to prevent stale responses. Do not cache a tool action's success response as though that action executed again. Consider an intent classifier, entailment check, or model based verifier for borderline matches, with an explicit cost/latency trade off. Monitor changes when embedding models or vector indexes are upgraded; old thresholds may no longer be calibrated. 5. What about production metrics? A cache with 70% hit rate but 5% incorrect hits may be unacceptable; another with 35% hit rate and almost no incorrect hits may be superior. Report uncertainty and slice false hits by intent, tenant, language and knowledge freshness. There is no single threshold that guarantees safety. Interview ready answer: “I'd label representative query–cached answer pairs, sweep similarity thresholds, and choose the point on the precision/recall versus latency/cost curve that meets an explicit incorrect hit budget. Hard tenant and context filters, TTL and source invalidation apply regardless of score. I'd validate in shadow mode and monitor wrong answer incidents after release.” References: Redis semantic cache documentation; Redis LangCache threshold guidance; Redis analysis of semantic cache evaluation.

How do cache invalidation and TTL prevent stale responses in LLM semantic caching?

Time to live (TTL) limits how long an answer stays eligible for reuse; cache invalidation explicitly removes or rejects an answer when its source, permissions, or assumptions change. Both matter because semantic similarity cannot establish factual freshness. TTL bounds staleness duration but cannot guarantee that an answer remains correct until expiry . 1. TTL versus invalidation TTL: an entry created at 10:00 with a 60 minute TTL expires at 11:00. A matching question at 10:30 may get the cached answer; at 11:01, it must be recomputed or refreshed. Event driven invalidation: when a policy changes at 10:12, affected answers should be invalidated then , rather than waiting until 11:00. Version based invalidation: attach a source document version, model/prompt version, tenant, locale and authorization scope; a cache hit is valid only for the appropriate versions. Memory eviction is different: it removes entries to recover storage even when they have not expired. 2. Beginner friendly numerical example A customer support chatbot caches: “Returns are allowed within 30 days.” 09:00: the answer is approved and cached with a 24 hour TTL . 10:00: the company changes its policy to 14 days . 10:05: without invalidation, the cached answer might incorrectly say 30 days for almost another 23 hours . 10:01 or later, upon receiving the change event: a properly coordinated invalidation removes the old policy's eligible entry (or switches the active policy version). A new query fetches the current 14 day policy and builds a fresh approved answer. Even a one hour TTL alone would leave a possible incorrect answer window. No TTL duration solves immediate consistency on its own. Times and TTLs are illustrative. 3. What should be stored with a cache entry? Question embedding and vetted response . Source identifiers and versions , such as policy revision or retrieval index snapshot. Created at and expires at timestamps, plus tenant/locale and required authorization boundaries. Model, prompt and safety policy versions if changes can affect response validity. Optional validation state and incident flags for disabling suspect entries. 4. Common invalidation strategies 1. Expiration: let short lived entries expire naturally, particularly for relatively stable FAQ content. 2. Targeted deletion: when a document or policy changes, delete cache entries associated with the changed source. Maintain dependency metadata rather than guessing based on text similarity. 3. Versioned namespaces: treat cache entries from a prior knowledge base version as ineligible. This avoids serving old answers even if physical deletion is delayed. 4. Immediate bypass: for live transactions, account specific records or urgent policy changes, bypass semantic response caching until updated data and authorizations are verified. 5. Refresh on miss: recompute through the usual LLM/retrieval path, validate, and store with the appropriate TTL. Ensure concurrent cache misses don't trigger excessive duplicate calls (cache stampedes). 5. Choosing TTLs Static educational explanations: often tolerate relatively long TTLs with version checks. Policies and product details: choose TTL based on update frequency, but invalidate immediately on authoritative changes. Prices, inventory, current events and personal account data: use very short TTLs, strict per user context rules, or disable response reuse entirely. Do not copy a universal TTL number: assess the cost of a stale error , how quickly changes can be detected and how reliably invalidation signals arrive. 6. Production monitoring Track cache hit rate, expired hits rejected, invalidation propagation time, stale answer incidents, false matches, p95 latency, LLM calls saved and eviction rate . Test a real change event: update a source, confirm the old answer cannot be served, and verify the next eligible request gets an updated response. Interview ready answer: “TTL puts an expiry deadline on stored answers, while event driven or version based invalidation removes eligibility as soon as underlying information changes. I'd tag cached answers with source versions and tenant context, invalidate dependencies on updates, and reserve short TTLs or bypasses for fast changing information. I'd measure stale answer incidents rather than optimizing only hit rate.” Research references: Redis semantic caching; Redis targeted cache invalidation; Amazon ElastiCache semantic caching best practices.

How do you prevent cache stampedes and concurrent duplicate LLM requests?

A cache stampede (thundering herd) happens when many requests simultaneously miss an expired or absent entry and each independently triggers expensive LLM generation. The standard defense is request coalescing (singleflight) : for an eligible exact cache key, allow one request to refill and let others await its result. Combine this with bounded waiting, safe fallback, rate limits, and carefully planned cache expiry. 1. Why it happens A popular answer expires or an application starts with an empty cache. Fifty users ask the same FAQ within a short interval. Without coordination, all fifty observe a miss, then launch fifty equivalent LLM calls. This can saturate inference concurrency, increase queue latency and multiply token costs. A semantic cache similarity match is not by itself a safe coalescing key : different questions with superficially similar embeddings may need different answers. 2. Beginner friendly numerical example Suppose 100 near simultaneous, identical and authorized requests each need an LLM call costing $0.02 and taking 2 seconds. Without coalescing: 100 calls, $2.00 in model charges, and 100 expensive generations competing for capacity. With correct singleflight: 1 underlying call, $0.02 in model charges, and up to 99 waiters reuse its verified result, a 99% reduction in duplicate generation calls . The 99 waiters still incur waiting time and cache/coordination overhead. Overall latency and total costs are not automatically reduced by 99% . These numbers are hypothetical. 3. End to end implementation 1. Build an exact safe key from canonicalized request inputs plus tenant/authorization scope, model, system prompt, retrieval/document versions, relevant tool context and generation settings. Never coalesce calls across incompatible contexts. 2. Check the cache. On a hit, enforce freshness and permissions before returning the stored answer. 3. On a miss, elect one owner of the in flight generation for this key. Other callers join its pending result rather than issuing duplicate model requests. 4. Publish carefully: verify the completed response, check that source versions and permissions still match, and then populate the cache. Wake waiters. 5. Handle failure: impose deadlines, limit waiter count, retry with backoff and jitter if appropriate, and release the ownership marker on success or error. A failed first request should not trap all waiters forever. 4. One process versus many replicas Single process: use an in memory per key future, promise or lock (often called singleflight ). This deduplicates work only within that process. Multiple replicas: consider a shared short lived lease or distributed lock (e.g. Redis), plus a shared result cache and notifications or bounded polling. A local lock in each replica does not prevent cross replica duplicate LLM calls. Safe lease handling: use an owner identifier, expiry, and conditional release; stale owners must not overwrite a newer result. Fencing/version checks matter when correctness is important. Do not assume lock TTL means the owner remains valid until its work ends. 5. Other complementary protections TTL jitter: stagger expiration across different keys so thousands of entries do not all expire together. It does not solve concurrent misses for one hot key. Stale while revalidate: briefly serve an older response while one worker refreshes only if the answer is still authorized and safe to serve stale . Never use this to bypass urgent policy or permission invalidation. Proactive refreshing: refresh popular, approved entries before TTL expiry while respecting rate limits. Admission controls: cap concurrent LLM calls and waiting requests; use backpressure rather than allowing unbounded memory/queue growth. Streaming: sharing one live output stream among callers needs special care for cancellation, backpressure, tool side effects and privacy; do not blindly merge personalized or action taking requests. 6. Production monitoring Track cache miss bursts, duplicate generation suppression ratio , singleflight waiters, lock contention and timeouts, generation error rates, queue depth, p95/p99 TTFT and cost per successful response. Test simultaneous requests, leader crashes, lease expiry, source version changes and tenant boundaries. Interview ready answer: “A stampede occurs when concurrent misses launch duplicate LLM work. I'd use per key singleflight for identical authorized inputs, with a shared lease across replicas where needed. TTL jitter spreads expiry across keys; safe stale while revalidate and proactive refresh reduce misses. I'd bound waiting and handle owner failures while preventing obsolete results or cross tenant data reuse.” Research: Redis cache layer architecture guide; Redis distributed locks; AWS CloudFront stale while revalidate.

How do you design a multi-level caching strategy for production LLM applications?

Use a cache hierarchy that avoids expensive work in the safest order: exact approved response reuse, then carefully gated semantic response reuse, then model inference with prompt prefix/KV caching. These layers store different artifacts and are not interchangeable. 1. The layers and what they cache Layer 1 — exact response cache: a validated complete answer keyed by the full canonical request, tenant/authorization scope, knowledge version, prompt/model settings, locale and any relevant tool state. A hit can return the answer without an LLM call. Layer 2 — semantic response cache: approved answers indexed by embeddings, reusable only when meaning, task intent and all hard context filters match. A high similarity score alone is insufficient. Layer 3 — prompt prefix/KV caching: model side attention key/value states for identical eligible prompt prefixes. A hit skips some prefill computation, but the model still generates a fresh response . Layer 4 — ordinary within request decode KV cache: stores previous attention states as tokens are generated, preventing repeated attention state recomputation for the same sequence. It is part of normal efficient autoregressive serving. 2. End to end request path 1. Authenticate the caller; resolve tenant, permissions, document versions and freshness requirements. 2. Exact cache lookup: return only an authorized, unexpired response valid for the complete execution context. 3. If eligible, semantic cache lookup: apply hard filters before vector search; enforce calibrated similarity thresholds and response validity checks. 4. Otherwise invoke the LLM or retrieval pipeline. Keep stable system instructions, tool definitions and shared documents at the start to maximize prefix cache reuse. 5. Validate and cache the final result only when safe , associating source versions, TTL, tenant scope and any required quality metadata. 6. Use singleflight/request coalescing for identical safe misses, plus bounded waiting and failure handling. Do not replay tool actions from a cached success message. 3. Beginner friendly numerical example Suppose 1,000 FAQ requests arrive: 300 exact cache hits: no model calls. Of the 700 remaining, 200 pass semantic matching and strict validation: no model calls. The remaining 500 require generation; 300 of these share a valid model side prompt prefix and benefit from KV reuse during prefill. Total model calls = 500 , versus 1,000 without application response caches: 50% fewer model invocations . Prefix caching additionally avoids some eligible prefill work on 300 model calls , not 300 further model calls . It does not make total calls 200. All figures are illustrative; cache lookup overhead, embeddings, eviction, provider token billing and output generation determine realized savings. 4. Correctness and security rules Never serve cross tenant or cross permission responses unless the content is explicitly public and the reuse policy permits it. Cache keys or filters must reflect model, tool, prompt and source versions when those factors can change the answer. Use TTL + event driven/version based invalidation . TTL alone can leave stale answers between an authoritative change and expiry. Do not use semantic response caching for personalized balances, live transactions, rapidly changing facts or side effecting tool calls without a strong, purpose built correctness protocol. Treat approximate semantic matches as a quality risk ; measure wrong answer rates, not only hit rate. For self hosted KV caching, use compatible execution contexts and isolation mechanisms; provider managed prompt caching has its own retention and billing rules. 5. When should each layer be enabled? High volume stable FAQs: exact cache + carefully validated semantic cache are usually effective. Long shared reference prompts with unique user queries: prefix caching is valuable even if response cache hit rates are low. Multi turn chat: the repeated conversation prefix can often benefit from provider or self hosted KV reuse; personalized complete answer caching may be inappropriate. Sensitive or time critical questions: bypass response caches as needed; allow only permitted model side prefix reuse within the supported security boundary. 6. Operations and monitoring Track cache hit rates per layer , correct response cache hit rate, stale/incorrect hits, embedding/search overhead, prefill TTFT, p95/p99 end to end latency, generated token volume, LLM requests avoided, KV occupancy/evictions, model goodput and cost per correct answer. Run load tests with repeated and unique prompts, version changes, expired hot keys, concurrent misses and tenant isolation. Interview ready answer: “I'd first attempt an exact approved response cache hit, then a policy gated semantic response cache hit. On a miss, I'd call the LLM with stable prompt prefixes so the inference engine can reuse KV states. Response caches can skip generation; prefix caches only skip eligible prefill. Every layer needs version aware keys, permissions, expiry and measured correctness. I'd compare cost per correct answer and tail latency, not just hit rates.” Research sources: Redis semantic caching; RedisVL response caching; vLLM Automatic Prefix Caching; OpenAI prompt caching.

How do you benchmark cache hit rates, latency, and cost savings in production LLM systems?

Benchmark caching against an uncached baseline using identical, production like traffic. A high hit rate alone is not success: the useful outcome is lower cost and latency per correct, authorized response , without more stale or incorrect answers. Measure response caching and model side prefix caching separately because they save different work. 1. Choose metrics for each layer Exact/semantic response cache request hit rate: eligible requests served from the response cache ÷ all eligible requests. Break this into exact hits, semantic hits, misses, and bypasses. Correct hit precision: approved, context valid answers ÷ response cache hits reviewed. A semantic hit that returns a wrong or unauthorized answer counts as a failure, even if it saves a model call. Model calls avoided: request count from which generation was safely skipped. Do not include prefix cache hits here. Prefix KV cache token hit rate: reusable cached prompt tokens ÷ eligible prompt tokens queried. This is token based, not response request hit rate. Latency: p50/p95/p99 end to end response time, time to first token (TTFT), decode/inter token latency, and cache lookup overhead. Cost: API uncached input, cached input, cache write and output token charges; embeddings, vector search, storage, GPUs, and operational costs. Use the actual provider/model tariff , not a generic discount. 2. Beginner friendly numerical example Consider 1,000 incoming requests : 200 exact response hits and 150 validated semantic response hits : 350 requests bypass LLM generation. 650 LLM calls remain . Of these, 400 use a compatible prompt prefix that benefits from KV reuse. Response cache hit rate = 350/1,000 = 35%. Model calls avoided = 350 . Prefix cache assisted model calls = 400 , but these calls still generate outputs. Do not subtract them again from 650. Suppose an uncached LLM call costs a hypothetical $0.02 , and 400 prefix assisted calls instead cost $0.014 while 250 remaining calls cost $0.02: No caches: 1,000 × $0.02 = $20.00 . Model charges with caches: 400 × $0.014 + 250 × $0.02 = $10.60 . If embedding, cache storage, lookup and operations cost another $1.40 , total = $12.00 . Net saving = ($20 − $12)/$20 = 40% or $8.00 per 1,000 requests. All prices, hits and savings above are invented for teaching. This also assumes the cached responses are correct and no other expenses change. 3. How to run a fair benchmark 1. Capture a representative traffic mix: prompt/output lengths, repeated versus unique prefixes, tenant distribution, task types, arrival rates and concurrency. Use authorized anonymized or synthetic traces. 2. Keep constant: model/checkpoint, sampling configuration, prompt templates, hardware, request SLOs, retrieval version and load pattern. 3. Compare no response caching/no prefix reuse , prefix caching only , exact response caching , exact + semantic response caching , and all layers together . 4. Run both cold cache and warm cache tests. Track steady state results separately from startup and cache refresh bursts. 5. Evaluate freshness and correctness on labeled queries, including paraphrases requiring different answers, expired entries and cross tenant boundaries. 6. Measure under realistic concurrency . High offline cache hits may not translate to low p99 latency if cache lookups, locks or model scheduling saturate. 4. Practical observability examples In vLLM, query the deployed version's Prometheus prefix cache hit/query counters , KV cache usage, active/waiting requests, and measured prefill/decode rates. Counter ratios over the same interval are preferable to mixing cumulative and windowed statistics. With OpenAI's API, inspect token usage including cached input tokens and cache write tokens where provided , then apply the model's published cached read/write and ordinary token prices. Provider accounting and retention rules vary by model. For Redis/semantic response caches, instrument cache lookups, validated hits, misses, expired entries, incorrect match incidents, and embedding/vector search latency at the application layer. 5. Common benchmarking mistakes Claiming a 70% request hit rate when the metric actually means 70% of input tokens reused. Treating cached prompt tokens as avoided model output tokens. Counting an unverified semantic match as a correct hit. Ignoring cache writes, lookup overhead, eviction, cache stampedes, API charges and cold starts. Reporting mean latency only while p95/p99 deteriorates. Comparing different user populations or changing model/prompt settings between A/B arms. 6. Decision rule Deploy a cache layer when controlled experiments show lower total cost per correct answer, acceptable p95/p99 latency, stable accuracy and freshness, and no authorization regressions . Break results out by layer and by workload; some traffic benefits from prefix reuse but is unsuitable for semantic response caching. Interview ready answer: “I’d benchmark cold and warm caches against the same uncached traffic and separate response level hit rates from token level prefix reuse. I'd measure correct hit precision, TTFT, p95/p99 latency, model calls avoided, actual input/cache write/output billing, and infrastructure overhead. The objective is cost per correct response under a latency SLO—not the biggest hit rate percentage.” References: vLLM Metrics; OpenAI Prompt Caching; Redis Semantic Cache.