Retrieval-Augmented Generation

Retrieval-augmented generation connects a generative model to a searchable knowledge source. Instead of relying only on model weights, the system fetches relevant documents and places them into the prompt or context.

How do you shard a large RAG index?

Sharding a RAG index means dividing the indexed vectors and their metadata into smaller partitions (shards), typically distributed across multiple search nodes. Each shard holds only part of the corpus, so the system can grow beyond one machine's storage, RAM, and query capacity. Beginner friendly example A company stores 80 million vectors . Assume four balanced shards: Shard A: 20 million vectors Shard B: 20 million vectors Shard C: 20 million vectors Shard D: 20 million vectors These counts are illustrative: real shards vary in size and workload. A query may be routed to one relevant shard or searched across all four, with partial results merged into the final Top K. Step by step design 1. Estimate the workload: Count vectors, raw and indexed bytes, embedding dimensions, filter usage, update rate, peak queries per second, and latency targets. 2. Choose the partition key: Use balanced hash partitioning for general searches, or a controlled tenant/region/time routing scheme when queries can be restricted to specific partitions. Avoid creating a separate physical shard for every tiny tenant. 3. Size shards for the engine: Keep each shard small enough to index, search, recover, and transfer within acceptable resource budgets. Balance shard size against excessive coordination overhead. 4. Distribute across nodes: Place shards to spread RAM, disk, CPU, and expected query traffic. Avoid putting every replica on the same failure domain. 5. Route requests: Apply authorization and eligible shard filters first. Search selected shards in parallel when appropriate. 6. Merge results: Each shard returns candidate IDs and similarity scores. The coordinator deduplicates and merges compatible scores or reranks a combined pool before passing cited evidence to the generator. 7. Replicate for availability: Sharding divides data; replication creates additional copies to survive failures and increase read capacity. A replica is not an additional unique shard. 8. Plan rebalancing and updates: Monitor hot shards, skew, ingestion lag, and shard size; move or reshard data safely while maintaining document IDs, permissions, and source versions. Small numerical sizing example Suppose 80 million vectors , with 768 float32 values per vector: Raw vector payload: 80,000,000 × 768 × 4 = 245.76 GB. Four balanced shards: approximately 61.44 GB raw vectors per shard . Two copies of each shard: 491.52 GB raw vector payload cluster wide . These figures exclude ANN structures, text, metadata, memory overhead, network cost, spare capacity, and backups. The actual number of nodes depends on those additional costs. Common partitioning strategies Hash by stable vector/document ID: Usually spreads data evenly but may require querying every shard. Tenant aware routing: Can reduce query fan out for tenant specific requests, but needs strong authorization and protection against large tenants dominating a shard. Region or time partitioning: Useful when queries naturally target a region or time range, but may cause hot spots or uneven distribution. Key risks Excessive fan out: Every request hitting dozens of shards increases network and coordination cost. Poor balance: One oversized or busy shard dictates overall tail latency. Incorrect routing: Missing a relevant shard hurts recall; missing an authorization filter risks exposing data. Too many tiny shards: Metadata, replicas, scheduling, and maintenance overhead become expensive. Confusing shards with replicas: Shards scale data capacity; replicas provide redundancy and extra serving capacity. Interview takeaway: Choose a sharding strategy based on balanced capacity, selective query routing, secure filtering, parallel search, result merging, and safe rebalancing . Benchmark shard count and Top K recall before production rollout. Reference: Qdrant distributed deployment: https://qdrant.tech/documentation/distributed deployment/

How do you distribute retrieval across multiple nodes?

Distributed retrieval in RAG runs searches on multiple retrieval nodes and combines their results before generating an answer. Unlike sharding—which decides where index data lives —distributed retrieval focuses on query execution, coordination, fault handling, and globally correct result ranking . Beginner friendly example Imagine 12 million chunks spread across 3 search nodes , each owning 4 million. A user asks: “Which deployment guide explains rolling restarts?” 1. A query coordinator authenticates the user, embeds the query, and applies access constraints. 2. The coordinator sends the query to the relevant nodes in parallel. 3. Each node searches its local index and returns its top 10 authorized candidates with document IDs and similarity scores. 4. The coordinator merges up to 30 candidates , deduplicates repeated documents, and selects or reranks the global top 5 . 5. The generation service receives only the strongest source passages, then answers with citations. This is a simplified flow. Returning only five from each node could exclude useful global candidates when deduplication or a later reranker changes the ordering. Key components Coordinator/router: Selects shards and replica targets, handles authentication, sends parallel requests, and maintains a query deadline. Search workers: Hold or access local vector and keyword indexes and produce ranked local candidates. Replica selection: Chooses an available copy of each required shard for redundancy and load balancing. Aggregation layer: Deduplicates, merges compatible scores, applies fusion/reranking, and enforces final evidence limits. Telemetry: Records per node latency, timeouts, selected shards, result counts, and final answer quality. Important design decisions 1. Selective routing vs scatter gather: Search one tenant/region shard when valid routing allows it; otherwise fan out to every relevant shard. 2. Preserve retrieval recall: Fetch sufficient candidates per shard. Final global Top K can change after reranking or merging keyword and vector results. 3. Compare scores correctly: Raw scores from incompatible embedding spaces or search methods cannot simply be mixed. Normalize/calibrate or use rank fusion and reranking. 4. Handle stragglers: Set timeouts and latency budgets, use load aware replicas, and avoid waiting indefinitely for a slow node. 5. Define partial result behavior: If a shard fails, a partial answer might be acceptable for low risk discovery, but should be labeled incomplete. For compliance or safety critical questions, fail closed rather than hiding missing evidence. 6. Enforce access control: Apply tenant and document permissions at each search node, not just after aggregation. 7. Keep indexes consistent: Coordinate versioned updates, deletions, shard placement, and replica lag. Stale replicas can return outdated evidence. Numerical latency example Suppose three nodes finish their searches in 35 ms, 50 ms, and 90 ms . With parallel fan out, retrieval completion is roughly driven by the slowest required node (90 ms) , plus network and aggregation overhead—not the sum of 175 ms. Actual production p95 latency also depends on scheduling, retries, and coordinator load. How to evaluate it Load test different shard counts and replica placements using realistic traffic. Track global recall@K , p95/p99 latency, queries per second, network fan out, missing shard rates, duplicate suppression, and end to end grounded answer quality. Interview takeaway: Distributed retrieval means route → search nodes in parallel → merge and rerank → handle failures → ground the answer . Sharding supplies the partitions; query orchestration turns them into a reliable search service.

How do you handle multi-tenant data isolation in RAG?

Multi tenant data isolation in RAG ensures that each organization (tenant) can retrieve and generate answers using only the documents it is authorized to access . The isolation rule must apply throughout ingestion, indexes, retrieval, caches, logs, and answer generation—not just in the user interface. Beginner friendly example A SaaS support assistant serves Company A and Company B . Both upload a document titled “Pricing Policy.” An employee of A asks: “What is our discount limit?” Retrieval must consider only A's authorized policy , even if B's document has a higher vector similarity score. The answer may cite A's policy but must never expose B's prices, passages, or metadata. Common architecture options 1. Separate indexes or collections per tenant: Stronger operational separation and simpler tenant scoped queries, but many small tenants may create expensive index management overhead. 2. Shared index with mandatory tenant filters: Store a trusted tenant id on every chunk and apply it in the search engine before ranking or returning results. Efficient for many tenants, provided enforcement is reliable. 3. Hybrid isolation: Use dedicated indexes for larger or regulated tenants and a carefully partitioned, filtered index for smaller ones. The right design depends on tenant count, scale, contractual rules, costs, and the vector engine's filtering guarantees. How to implement safely 1. Authenticate the user: Derive tenant identity from validated credentials/session claims, not from a tenant ID supplied inside a chat message. 2. Authorize the request: Establish document level permissions, user roles, and any cross team sharing rules. 3. Tag data at ingestion: Attach tenant ID, document ID, permission scope, source version, and deletion state to every stored chunk and embedding. 4. Enforce retrieval restrictions: Filter authorized tenants and documents in the database or search service before evidence reaches the model. Avoid searching globally and trusting the LLM to ignore unauthorized matches. 5. Protect derived artifacts: Tenant scope retrieval caches, embeddings tied to private documents, reranker candidates, conversation history, and any generated summaries. 6. Verify downstream context: Ensure every passage provided to the LLM passes the same authorization policy, including during graph traversal and multi source search. 7. Audit and test: Log safe tenant scoped identifiers and denials. Test missing filters, forged IDs, cross tenant near duplicate documents, cache collisions, and permission changes. Small numerical example Suppose a shared vector collection has 1 million chunks , with 400,000 for A and 600,000 for B . An A user's search for Top K = 5 must return up to 5 authorized results from A's eligible 400,000 , not the global top five with B's results removed afterward. The latter can reduce recall and, if filtering is misplaced, risk unauthorized exposure. Common mistakes Trusting an LLM prompt such as “only use Company A documents” as a security boundary. Using an untrusted tenant id parameter without validating ownership. Filtering only after retrieval or generation. Reusing cached contexts across tenants. Updating permissions in the primary database but leaving stale vector index access rules. Assuming tenant isolation alone handles finer user or document level authorization. Interview takeaway: Multi tenant RAG must enforce a trusted tenant boundary at every retrieval and data processing layer . Shared indexes can work, but tenant filters and permission checks must be server enforced, testable, and fail closed.

How do you handle PII in a production RAG system?

Handling personally identifiable information (PII) in production RAG means protecting information that can identify a person—such as names linked to account records, email addresses, phone numbers, identification numbers, and sensitive customer details—throughout ingestion, retrieval, generation, logging, and deletion. Beginner friendly example A support assistant indexes customer tickets. One ticket contains: Customer: Maya Rao; email: maya@example.com; order 742; refund pending. A general question such as “What are common refund delays?” should use only the relevant issue summary, not expose the customer's identity. An authorized agent investigating order 742 may access necessary fields, but only after authentication and permission checks. Practical production controls 1. Inventory and classify data: Identify PII in uploaded PDFs, tickets, chat transcripts, tables, metadata, and images. Include indirect identifiers when combined data could identify an individual. 2. Minimize before indexing: Remove or mask unnecessary fields at ingestion. For example, store [CUSTOMER EMAIL] instead of a raw email if semantic retrieval does not need it. 3. Use reversible tokenization only when needed: Replace identifiers with protected tokens and keep the re identification mapping in a separate, access controlled system. Hashing alone may not protect predictable values. 4. Encrypt and restrict storage: Protect source documents, embeddings, text chunks, caches, and backups with appropriate encryption, access controls, and lifecycle rules. 5. Enforce access during retrieval: Apply tenant, role, user, document, and purpose based restrictions before returning passages to the model. A prompt telling the model to keep secrets is not an authorization boundary. 6. Limit what reaches the LLM: Redact unnecessary PII from retrieved context and verify the model provider's retention, processing, region, and contractual settings. 7. Check generated responses: Detect unauthorized disclosure of personal data; apply policy based redaction or block the response. Do not rely on output filtering as the only control. 8. Protect observability: Avoid logging full prompts, raw search results, user identifiers, or model outputs by default. Restrict and expire any sensitive traces. 9. Support correction and deletion: Track source to chunk mappings to propagate edits and removals across indexes, caches, derived summaries, and applicable backups according to policy. 10. Test the boundaries: Include adversarial questions, cross tenant attempts, prompt injection, obsolete permissions, and accidental PII in citations or snippets. Small numerical example Suppose 10,000 support tickets contain 2,000 email addresses and 600 phone numbers . A preprocessing policy detects and masks the email and phone fields before embedding. The searchable knowledge base can still retain issue descriptions for FAQ retrieval while authorized customer specific workflows use the protected system of record. The counts are illustrative; masking accuracy must be validated with a labeled sample. Common mistakes Assuming embeddings are automatically anonymous or impossible to extract sensitive information from. Removing PII from visible text but leaving it in metadata, OCR output, logs, or caches. Applying redaction only after the LLM has already received restricted content. Treating a user supplied customer or tenant ID as proof of access. Forgetting that privacy requirements depend on jurisdiction, data purpose, and contractual obligations. Interview takeaway: PII protection in RAG requires data minimization + server enforced authorization + protected storage/model inputs + leakage checks + traceable deletion across the complete retrieval lifecycle.

What should be cached in a RAG system?

Caching in RAG stores reusable results from expensive pipeline steps so repeated or similar requests can be answered with less latency and cost. The safest cache candidates are stable, versioned, and appropriately permission scoped artifacts . Not every answer should be cached. What can be cached? 1. Document processing results: Parsed text, OCR output, chunks, extracted metadata, and image captions. Key entries by source document version and preprocessing configuration. 2. Document embeddings: Reuse vectors for unchanged chunks. Key by chunk content hash and embedding model/version so a model migration does not silently reuse incompatible vectors. 3. Query embeddings: Identical normalized queries encoded with the same embedding model can reuse an embedding, provided tenant/privacy policies permit. 4. Retrieval results: Cache ranked document IDs or passages for common questions; include index version, filters, authorization scope, and retrieval settings in the key. 5. Reranking outputs: Reuse a query–candidate ranking if the query, candidate set, reranker model, and scoring settings are unchanged. 6. Prompt prefixes or context: Some model providers support caching repeated prompt prefixes, such as stable instructions or shared documentation. Respect provider specific cache semantics. 7. Final answers: Cache only where facts are stable, authorization is handled, and answers can be invalidated when source content changes. Prefer short lifetimes for frequently updated or personalized answers. Beginner friendly example A company's help desk receives 1,000 questions per day . Suppose 300 ask the identical question, “How do I reset my password?” Without query level caching, those 300 requests may repeat retrieval and generation. With a valid cache entry, the system could reuse the answer or retrieve cached evidence for up to 299 follow up requests . If the reset procedure changes, the cache must be invalidated; otherwise, the fast answer may be wrong . This is an illustrative upper bound, not a guaranteed hit rate or cost saving. How should cache keys be designed? A retrieval cache key may contain: hash(query + tenant id + user access scope + index version + filters + retriever config) An embedding cache key may use: hash(normalized text + embedding model version) Do not use a shared retrieval or answer cache across users with different document permissions. Content hashes also need careful handling for confidential text; cache storage must enforce its own access controls. Invalidation rules Source updates or deletions: Invalidate entries that depend on changed documents, using source version IDs and dependency tracking where practical. Permission changes: Invalidate or re authorize cached evidence before serving it. Pipeline changes: Refresh caches when embedding models, retrieval/reranking settings, prompts, or generation models change. TTL: Use time to live as a fallback, not the only freshness control for sensitive or frequently changing knowledge. Observability: Monitor hit rate, stale hit incidents, time saved, token savings, cache memory, and cross scope access attempts. Trade offs Caching improves latency and cost but adds invalidation complexity, storage overhead, and privacy risk. A cached answer that bypasses current authorization or uses deleted evidence is worse than a slower fresh retrieval. Interview takeaway: Cache stable preprocessing artifacts, embeddings, retrieval/reranking outputs, prompt prefixes, and sometimes final answers —but include versioning, permissions, and invalidation in the design from the start.

What is embedding caching?

Embedding caching stores the vector representation produced by an embedding model so an identical, unchanged text input does not have to be embedded again. It is useful during both document ingestion and query processing in retrieval augmented generation (RAG). Beginner friendly example An internal knowledge base contains 10,000 document chunks. During a nightly sync, only 200 chunks change. Without an embedding cache: The system may recompute all 10,000 embeddings. With a valid cache: It reuses embeddings for the 9,800 unchanged chunks and computes only 200 new embeddings . Illustrative savings: 9,800 fewer embedding operations, or 98% of that batch. This does not mean retrieval or generation is cached—the reused vectors still participate in search. How does embedding caching work? 1. Normalize input consistently: Preserve meaning while applying a defined text normalization policy. 2. Create a cache key: Include a hash of the exact model input, embedding model identifier/version, and relevant preprocessing or configuration settings. 3. Look up the key: If a valid vector exists, reuse it. 4. On a miss: Call the embedding model and cache the resulting vector together with its model metadata and dimension. 5. Invalidate safely: Recompute when text, preprocessing, model version, or embedding configuration changes. For example: cache key = hash(model id + model version + preprocessing version + exact input text) Do not reuse vectors from incompatible embedding models merely because they have the same dimensions. Document embeddings vs query embeddings Document embedding cache: Avoids re embedding unchanged source chunks during re ingestion or indexing. Query embedding cache: Avoids repeat encoding of identical user queries. Retrieval cache: Stores search results (IDs and rankings), not just the query vector; it is a different optimization. Important security and freshness considerations Treat cached embeddings and their metadata as potentially sensitive; embeddings do not guarantee anonymity. Apply tenant appropriate isolation and access policies when storing or reusing private data. Handle source deletions and retention requirements in the cache, index, and backups. Check the vector dimension and compatibility with the current retrieval index. Track hit rate, avoided calls, cache storage, and embedding latency, alongside changes in answer quality. Common mistakes Using the raw question alone as a key while changing embedding models. Applying aggressive text normalization that changes the meaning. Confusing embedding cache hits with retrieval cache hits. Keeping cached private embeddings indefinitely after source data is deleted. Expecting caching to improve semantic relevance; it mainly improves latency and cost for repeat work. Interview takeaway: Embedding caching is content and model version aware reuse of computed vectors . It reduces repeated embedding calls while preserving retrieval correctness through compatible keys, lifecycle management, and access controls.

What is retrieval caching?

Retrieval caching in RAG saves the results of a retrieval request—such as ranked document IDs, chunk references, scores, and sometimes authorized snippets—so a repeated request can avoid running the same vector or keyword search again. Beginner friendly example An employee asks: “What is the leave carry forward policy?” The retriever searches the HR index and returns passages from the current policy. If another authorized employee with identical access asks the same question while the index and rules remain unchanged, the system can reuse the retrieved candidates instead of searching again. Retrieval caching is different from embedding caching: An embedding cache reuses a query vector; a retrieval cache reuses the search results produced from that vector . Neither automatically caches the LLM generated answer. How does it work? 1. Build a safe cache key: Include normalized query or query embedding identity, tenant and effective authorization scope, index/version, metadata filters, retriever settings, and relevant ranking configuration. 2. Check for a valid hit: Return cached candidate references only if the entry remains fresh and authorized. 3. On a miss: Run the normal search and store the result with expiration and dependency information. 4. Revalidate before generation: Recheck document permissions, deletion status, and source versions; fetch current passages when storing IDs rather than raw snippets. 5. Invalidate on change: Purge or version entries when documents, permissions, filters, index contents, or retrieval models change. Example key: hash(query + tenant scope + permissions version + index version + filters + retriever config) The exact key depends on whether results are user specific and whether documents can change independently. Numerical example Suppose 1,000 searches arrive during one hour, and 300 repeat cacheable query and permission combinations. If all 300 hit valid entries, the backend performs 700 retrieval executions , saving 30% of searches. Actual savings depend on cache hit rate, entry lifetime, and validation overhead. When is it useful? High volume, repetitive FAQ searches with fairly stable documents. Expensive hybrid retrieval, multi shard fan out, or reranking steps. Tenant specific knowledge bases with predictable access rules. Important caveats Never bypass authorization: A globally cached Top K must not leak documents from another tenant or user. Avoid stale citations: Invalidate entries when an indexed document is updated or removed. Keep keys precise: Different filters, query rewrites, embedding versions, or access scopes may require distinct entries. Be careful with semantic caching: Similar looking questions may have different intent, dates, or entity constraints. Exact match caching is simpler to validate. Monitor correctness: Track hit rate, latency reduction, stale hit incidents, and permission failures—not only cost savings. Interview takeaway: Retrieval caching reuses ranked evidence candidates , improving query latency and search cost while requiring strict permission aware keys, versioning, and invalidation .

What is prompt or context caching in a RAG pipeline?

Prompt or context caching in RAG reuses previously processed input tokens or a stable prompt prefix when calling a language model. It reduces repeated model side processing of instructions or source passages, depending on the model provider's caching support. It does not mean the system skips retrieval or automatically reuses a generated answer. Beginner friendly example A policy assistant sends this prompt to an LLM: Stable prefix: System instructions plus 5,000 tokens of an unchanged HR policy. Variable suffix: The employee's new question. If 100 employee questions use the same exact eligible prefix , a provider supported prompt cache may reuse work for that prefix on later requests. The system still processes each new question and generates a new answer. Actual pricing, latency savings, minimum prefix size, cache lifetime, and matching rules depend on the model provider. How does it work? 1. Separate stable from changing content: Put reusable system instructions and authorized, versioned context before the question when supported by the model's cache mechanics. 2. Maintain consistent prefix content and order: Even small edits can invalidate or reduce cache hits, depending on the implementation. 3. Call the model with provider supported caching: Some APIs automatically cache eligible prefixes; others support explicit cache creation or controls. 4. Process the fresh suffix: The user's current question, request specific constraints, and retrieved evidence that changes must still be handled. 5. Record usage: Track cached input tokens, uncached input tokens, total latency, and spend. 6. Refresh safely: Changes to source passages, prompts, model versions, access permissions, or cache expiration may require a new prefix or cache entry. Numerical example Suppose 20 requests each have a 6,000 token prefix and a 200 token unique question . Without reuse: 20 × 6,200 = 124,000 input tokens processed across requests. After the first request, assuming all 19 later prefixes qualify for a cache hit: 6,200 + 19 × 200 = 10,000 uncached equivalent input tokens , plus 19 × 6,000 = 114,000 cached input tokens. The second figure is an illustrative accounting split, not a promise that only 10,000 tokens are billed . Cached tokens are often charged at a different rate rather than free. Prompt caching vs other RAG caches Embedding caching: Reuses vector representations of text. Retrieval caching: Reuses ranked document candidates. Prompt/context caching: Reuses eligible LLM input prefix processing . Answer caching: Reuses an entire previously generated response, with additional freshness and authorization risks. Important caveats Dynamic retrieval reduces prefix reuse: If every question retrieves different chunks or orderings, the prefix may change; stable instructions may still be cacheable. Security matters: Never reuse or expose confidential tenant specific context across unauthorized users. The provider's caching mechanics and isolation guarantees must be understood. Freshness matters: Cached processing of an old policy is not valid evidence after the source changes. Cache hits are provider specific: Exact matching, expiration windows, token thresholds, and model compatibility vary. Interview takeaway: Prompt caching speeds up repeated LLM calls by reusing stable input prefix computation . Design stable, authorized prefixes and measure cached tokens, latency, cost, and invalidation separately from retrieval caching.

How do you measure embedding latency in RAG?

Embedding latency in RAG is the time required to transform input text into an embedding vector. For user facing searches, it usually means query embedding latency ; document embedding latency is typically measured separately during ingestion. What exactly should be timed? 1. Query embedding: Start the timer just before requesting an embedding for the user's finalized search query and stop when the vector is available. Report whether measurements include network transmission, provider queueing, and retries. 2. Document embedding: Measure time to embed document chunks during ingestion, plus batch throughput. Do not mix this background workload into user facing retrieval timing. 3. Cache behavior: Measure cache hits and misses separately. A near zero cache hit should not be presented as model inference latency. 4. End to end RAG latency: Keep query embedding separate from vector search, reranking, prompt assembly, and LLM generation. Beginner friendly numerical example Suppose five uncached query embedding calls take: 40 ms, 50 ms, 45 ms, 65 ms, 100 ms Total time = 300 ms Average latency = 300 / 5 = 60 ms Sorted latencies = 40, 45, 50, 65, 100 ms Maximum = 100 ms With only five measurements, percentile estimates are unstable. For production p50, p95, and p99, collect a much larger, representative sample and state the percentile calculation method. Recommended instrumentation Record a trace or span with: Request/trace ID and model name/version Input token or character count Cache hit/miss Batch size, if relevant Duration in milliseconds Success, timeout, throttling, retries, and error status Region/provider or serving instance, when relevant Avoid logging raw sensitive queries by default. How do you measure production performance? Monitor p50, p95, p99 , mean, error rate, and traffic volume. Separate cold start from warmed up model behavior. Compare short and long queries, cache hits/misses, and concurrency levels. For remote embedding APIs, distinguish client observed latency from provider inference time where possible. Measure document ingestion throughput as chunks or tokens embedded per second, and track ingestion queue delay separately. Use consistent measurement boundaries when comparing embedding providers or model versions. Common mistakes Timing the entire RAG pipeline and calling it embedding latency. Mixing cache hits with misses without reporting the mix. Ignoring retries or rate limits that raise user perceived latency. Using only average latency while high percentile delays hurt user experience. Comparing models using different query sizes or different network conditions. Interview takeaway: Measure embedding latency as a clearly bounded query to vector operation , trace it independently, report meaningful latency percentiles, and monitor cache effects and ingestion throughput separately.

How do you measure retrieval latency in RAG?

Retrieval latency in RAG is the elapsed time spent finding candidate evidence from a searchable index or external knowledge source. Measure it separately from query embedding , reranking , and LLM generation so bottlenecks can be diagnosed correctly. Beginner friendly example A support assistant performs a vector search after embedding the user's question. The search begins when the retriever receives the query vector and filters, and ends when it returns the ranked candidates. Suppose five uncached search requests take 20, 30, 40, 50, and 110 milliseconds . Average retrieval latency: (20 + 30 + 40 + 50 + 110) / 5 = 50 ms . Median (p50): 40 ms . Maximum: 110 ms . With only five observations, p95/p99 estimates are unreliable; production percentiles need a much larger sample. How to instrument retrieval 1. Define boundaries: Start immediately before the retriever request and stop after the candidate list is received. State whether network transport and coordinator merging are included. 2. Trace substeps: In distributed retrieval, separately time routing, shard fan out, local ANN/keyword search, filtering, network transfer, and global merging. 3. Record query context: Capture retrieval method, index/version, number of shards searched, candidate Top K, filters, vector dimensions, and response status—not raw sensitive questions by default. 4. Separate cache hits: A cached result bypasses some search work. Record hit and miss latency separately, plus the combined user observed distribution. 5. Measure under load: Collect p50, p95, p99, throughput, concurrency, timeout rate, and error rate using representative production traffic. 6. Check quality alongside speed: Compare latency against recall@K or judged evidence relevance so a faster but less accurate index is not declared better. Numerical distributed search example Three shards respond in 25 ms , 40 ms , and 75 ms . If searched concurrently, the minimum retrieval completion time is roughly bounded by the slowest required shard, 75 ms , plus coordinator and network overhead—not the sum of 140 ms. Retries, a slow shard, or waiting for all replicas can raise tail latency. What commonly increases retrieval latency? Large candidate pools or high ANN search effort settings. Expensive metadata filters or poorly indexed filter fields. Many shards, network hops, overloaded nodes, or slow replicas. Hybrid search and additional result fusion stages. Cold disk/page caches and index updates or compaction. Rate limiting or retries when querying managed search services. Retrieval vs end to end timing Document the treatment of internal reranking or hybrid merge steps, because some search products include these within a single retrieval request. Interview takeaway: Measure retrieval latency with an explicitly bounded search span, break down distributed search work, track tail latency and quality, and compare cache misses under realistic load before optimizing the retrieval engine.