How do you measure retrieval latency in RAG?

Retrieval latency in RAG is the elapsed time spent finding candidate evidence from a searchable index or external knowledge source. Measure it separately from query embedding, reranking, and LLM generation so bottlenecks can be diagnosed correctly.

Beginner-friendly example

A support assistant performs a vector search after embedding the user's question. The search begins when the retriever receives the query vector and filters, and ends when it returns the ranked candidates.

Suppose five uncached search requests take 20, 30, 40, 50, and 110 milliseconds.

• Average retrieval latency: (20 + 30 + 40 + 50 + 110) / 5 = 50 ms.
• Median (p50): 40 ms.
• Maximum: 110 ms.

With only five observations, p95/p99 estimates are unreliable; production percentiles need a much larger sample.

How to instrument retrieval

1. Define boundaries: Start immediately before the retriever request and stop after the candidate list is received. State whether network transport and coordinator merging are included.
2. Trace substeps: In distributed retrieval, separately time routing, shard fan-out, local ANN/keyword search, filtering, network transfer, and global merging.
3. Record query context: Capture retrieval method, index/version, number of shards searched, candidate Top-K, filters, vector dimensions, and response status—not raw sensitive questions by default.
4. Separate cache hits: A cached result bypasses some search work. Record hit and miss latency separately, plus the combined user-observed distribution.
5. Measure under load: Collect p50, p95, p99, throughput, concurrency, timeout rate, and error rate using representative production traffic.
6. Check quality alongside speed: Compare latency against recall@K or judged evidence relevance so a faster but less accurate index is not declared better.

Numerical distributed-search example

Three shards respond in 25 ms, 40 ms, and 75 ms. If searched concurrently, the minimum retrieval completion time is roughly bounded by the slowest required shard, 75 ms, plus coordinator and network overhead—not the sum of 140 ms. Retries, a slow shard, or waiting for all replicas can raise tail latency.

What commonly increases retrieval latency?

• Large candidate pools or high ANN search-effort settings.
• Expensive metadata filters or poorly indexed filter fields.
• Many shards, network hops, overloaded nodes, or slow replicas.
• Hybrid search and additional result-fusion stages.
• Cold disk/page caches and index updates or compaction.
• Rate limiting or retries when querying managed search services.

Retrieval vs end-to-end timing

Question
  → query embedding        [embedding latency]
  → vector/keyword search  [retrieval latency]
  → reranker               [reranking latency]
  → LLM answer             [generation latency]

Document the treatment of internal reranking or hybrid merge steps, because some search products include these within a single retrieval request.

Interview takeaway: Measure retrieval latency with an explicitly bounded search span, break down distributed search work, track tail latency and quality, and compare cache misses under realistic load before optimizing the retrieval engine.