Your RAG response latency doubles after adding a third retrieval source. How would you diagnose the critical path and reduce fan-out overhead?
Production incident (illustrative): A SaaS-support RAG assistant searches two authorized sources: product manuals (A) and internal runbooks (B). Its full-answer p95 is 2.2 seconds. Engineering enables a third source, a connector-backed customer knowledge API (C), to increase coverage of niche cases. Full-answer p95 rises to 4.4 seconds, even for questions answerable entirely from A. A waterfall reveals A completes in 170 ms, B in 240 ms, and C usually completes in 320 ms but sometimes stalls for 2,100 ms. The orchestrator starts all three in parallel but waits for every response, then globally reranks 75 instead of 30 candidates and expands prompt tokens. “Parallel search” did not mean “constant latency”; both the slowest branch and downstream merge/rerank/generation work increased.
Core principle: Identify the end-to-end request critical path and separately account for (1) additional required fan-out dependencies, (2) connector tail latency, (3) selection/merge/rerank overhead, (4) context-token inflation and LLM output latency, and (5) concurrency, retries, connection pools and queuing. Optimize information gain per unit of critical-path latency, never source count alone. Research on *The Tail at Scale* shows that as a request waits for more dependent services, rare slow operations can dominate aggregate responsiveness; Microsoft’s RAG guidance describes parallel multi-source retrieval but notes that reranking and expanded context introduce additional latency/cost.
1. Establish an apples-to-apples regression
- Freeze a representative set of authorized question/answer tasks, query rewrite, model revisions, source document versions, tenant/ACL, top-k, prompt template, output-token cap, hardware, traffic level and cache-hit ratios. Compare A+B with A+B+C via paired replay and a feature flag.
- Segment by routing intent: (i) questions solvable using A/B, (ii) questions requiring C, (iii) mixed-source comparisons and (iv) ambiguous queries. A source that improves one segment may be an avoidable dependency in another.
- Define metrics for full-response p50/p95/p99, streaming time to first token (TTFT), answer correctness/grounding, per-source recall, unique evidence contributions, source-routing accuracy, cost, token use and failure/timeout fraction.
- Do not assume doubled p95 implies one source doubled its average duration. Capture trace timings at the same traffic/concurrency and compare paired or stratified percentile distributions, with confidence intervals and provider/source status codes.
- Determine whether observed “latency” is wall-clock until all retrieval completes, until first generated token, or until last streamed token. A faster retrieval span may not improve the user-visible completion time if generation dominates.
- Keep source C’s permissions intact in all tests. It is unacceptable to route around mandatory access-controlled evidence merely to achieve a latency target.
2. Instrument a per-request critical-path trace
- Parent spans:
rag.request→query.prepare→retrieval.fanout(child A, B, C) →merge.dedupe→rerank→context.pack→llm.generate→response.render. Recordstart_ns,end_ns,queue_wait_ms,service_ms,source_id,candidate_count,input_tokens,output_tokens, cache outcome, retries and content-version hash. - Annotate dependency edges, not just spans:
rerankrequires which sources? Doesllm.generatestart after all three, after a quorum, or after a sufficiency gate? Critical-path calculation requires knowing what blocks the next step. - For a true parallel all-required fan-out, retrieval wall-clock is approximately
max(T_A,T_B,T_C)+orchestration, notT_A+T_B+T_C. If the implementation accidentally performs sequentialawaitcalls before launching the next fetch, its wall-clock is closer to a sum. - For p95 diagnosis, avoid
p95(A)+p95(B)+p95(C)or even simplymax(p95(A),p95(B),p95(C))as an exact request p95: per-request correlation and tail dependencies matter. Derive end-to-end p95 from complete recorded requests. - Separate connection-pool and semaphore queue wait from actual remote service duration. Adding source C can exhaust the shared HTTP client pool, increase DNS/TLS connection creation or block A/B even if C is not itself on every query's critical path.
- Trace
429/503, retry count, redirect/auth refresh, connector rate limits, circuit status and cancellation. Nested retries can make C appear to have an inexplicable 2-second median. - Capture merge/dedup latency, fusion branches, global reranker candidates and token-budget utilization. Source C may contribute no useful answer evidence while forcing many more candidates through a costly cross-encoder.
- Correlate p95/p99 with cache misses, source C request count, large C payloads, fan-out width, tenant, region and prompt length. If only some queries regress, it is likely an eligibility/routing or tail problem rather than universal LLM slowness.
3. Locate the first regression boundary
- A. Query planning: inspect source-selection traces. Did the router start calling C on every question despite C containing only specialized content? Check routing false positives and whether the source is optional or mandatory for the task.
- B. Fan-out start: compare actual A, B and C
starttimes. If calls are launched serially, fix concurrent scheduling within bounded per-source limits; do not sequentially await A before launching B and C. - C. Synchronization: identify an all-of barrier,
gather, or framework node that waits for slow C, even when adequate A/B evidence already exists. A “parallel” implementation can still have tail latency equal to the slowest required source. - D. Infrastructure contention: measure HTTP connection pool queue, provider-specific semaphore, container worker count, CPU, memory, network and egress. If A/B got slower after C was enabled, C may be starving shared transport resources.
- E. Candidate explosion: if global candidates grow from 30 to 75, profile dedup, fusion and cross-encoder rerank. Guard against near-duplicate C passages crowding A/B evidence and increasing context tokens.
- F. Generation inflation: compare exact final prompt byte/token counts, truncation, model version, LLM time to first token and output length. Increasing evidence can double LLM prefill or token charges depending on model, even when retrieval remains fast.
- G. Cache behavior: check whether fan-out bypasses a previous A/B cache key or causes cold C lookups on every request; inspect cache invalidation scope and source version safety.
- H. Concurrency: use stepped load to see whether C saturates its own quota and triggers retries or pushes shared orchestrator concurrency over an admission limit. Q276 covers general capacity; Q277 covers embedding retry storms.
- I. Evidence value: compare
recall@k, per-questionC-onlygold-span coverage and correctness with/without C. If C supplies indispensable citations in 12% of questions, unconditional deletion is unsafe; optimize routing. - J. Regression attribution: use controlled A/B and A/B/C replay with exact evidence manifests, identical answer model settings and source-version pins. A change in grounding, citation support or protected exception coverage is part of the regression.
4. Controlled experiments that isolate overhead
- Experiment 1: three-way waterfall—record 1,000 paired requests before/after source C (same query mix and arrival pattern), with per-span wait/service times and answer-quality labels.
- Experiment 2: C disabled versus enabled—leave the reranker, context packer and generator configuration identical. If retrieval span increases sharply but generation token usage is constant, look at C/barrier/connection pools.
- Experiment 3: C stubbed—return a fixed, authorized canned C response at negligible latency in an isolated test. If p95 remains high, candidate processing and generation are likely major causes; if it falls, C transport/server/tail is implicated.
- Experiment 4: fixed fan-out, fixed candidates—feed a frozen A+B+C result set to fusion/reranking and compare 30 vs 75 candidate sizes; this isolates selection overhead from remote waits.
- Experiment 5: identical packed prompt—replay a fixed serialized LLM input, regardless of source count. A model delay that disappears under prompt pinning implicates extra tokens/content or provider load rather than “number of retrieval sources.”
- Experiment 6: selective routing—send A/B-sufficient questions to A+B only, and C-dependent questions to A+B+C. Measure router false negatives that could hide critical evidence.
- Experiment 7: tail injection—simulate a 2-second C delay and test wait-all versus an authorized, sufficiency-checked C-skipping policy. If C is required for the user’s answer, no latency optimization may silently drop it.
- Experiment 8: controlled concurrency—increase arrival rates and compare queue wait/connection usage to separate source C’s intrinsic slow calls from orchestration resource saturation.
- Use randomized interleaving/counterbalanced order for offline evaluation to avoid model/index drift, plus paired bootstrap over query IDs; monitor errors and source freshness, not only latency.
5. Beginner-friendly numerical production example
- Before adding C, each request has 150 ms embedding/query preparation, parallel retrieval A 170 ms and B 240 ms, 110 ms merge+rerank, 100 ms context packing, and 1,400 ms generation. Approximate full critical path:
150 + max(170,240) + 110 + 100 + 1400 = 2,000 ms(2.0 seconds). - After adding C, representative slow-path requests have A 170 ms, B 240 ms, C 1,250 ms (including connector queuing). Merge+rerank rises to 310 ms because candidates grow 30 → 75; context packing increases to 160 ms; generation increases to 1,950 ms because the prompt grows. Critical path:
150 + max(170,240,1250) + 310 + 160 + 1950 = 3,820 ms(3.82 seconds). A mild additional cold-start or request/response delay of 180 ms can produce 4.0 seconds, approximately double the previous 2.0 seconds. - Attribution for the 1,820 ms core difference: C changes fan-out barrier from 240 → 1,250 ms (+1,010 ms); reranking +200 ms; packing +60 ms; generation +550 ms. Thus the largest contributor is the C-dependent barrier, but downstream work accounts for 810 ms, so optimizing C alone cannot restore the 2-second baseline.
- Selective source value: suppose 1,000 labelled queries, of which 850 can be correctly answered with A/B and 150 require C for an authoritative fact. A router identifies 140/150 required-C queries (93.3% recall) while unnecessarily sending 90/850 A/B-only queries to C. Total C calls: 140+90=230 instead of 1,000, a 77% reduction in C call volume, but 10 C-required questions are missed and must be caught through uncertainty/refetch or improved router recall before rollout.
- Optimization: query-aware routing + capped per-source candidate depth + dedup before cross-encoder rerank yield an illustrative 2.35-second p95 over a defined mixed workload, compared with 4.4 seconds after unconditional C. That's a 46.6% p95 reduction, assuming source coverage/groundedness remains acceptable; these p95s are *separately measured aggregate hypotheticals*, not sums of per-stage p95s.
- Per-source tail risk: if C has independent p95 of 1,250 ms while A/B are ~200 ms, each mandatory C request still needs a realistic C-specific latency/quality trade-off. A source-aware deadline and partial-result contract are explored under #279, and strict cross-stage deadline allocation under #280.
- Counterexample: simply removing C restores fast answers but misses 150/1,000 = 15% of questions. That improvement in latency would be unacceptable if those questions are critical. Track C-only evidence recall, not only average p95.
- All quantities are illustrative, not observed MachinoAI timings. The deterministic sums depict individual hypothetical request paths, not percentile arithmetic; real p95 values require many actual traces.
6. Engineering solutions and rollout
- Query-aware source routing: predict candidate source(s) from intent, entity/product, region and authoritative scope. Make C optional for A/B-sufficient questions, but require it for known C-dependent categories; calibrate false negatives against high-severity evidence.
- Parallel but bounded fan-out: launch independent calls concurrently, with separate per-source concurrency limits and isolated HTTP connection pools where justified. Bound total calls/query, reduce duplicate decomposed searches, and share reused embeddings.
- Evidence sufficiency gate: after A/B results arrive, test whether they cover each required answer facet from an authoritative source. Only skip C when the result is sufficient *and* policy permits source omission; if uncertain, fetch C or disclose limited evidence.
- Two-phase retrieval: run fast source(s) first for common queries, then selectively invoke slower C on uncertain/complex cases. This saves calls but can increase C-required latency because C no longer starts in parallel. Compare staged versus eager parallel routing on source-dependent segments.
- Source-aware candidate budgeting: cap and prioritize C candidates by relevance, authority and unique fact contribution rather than blindly appending its top-k. Dedup before expensive reranking; retain required exceptions and multi-source coverage.
- Reranker capacity controls: batch efficiently and cap cross-encoder comparisons; use source-specific score calibration for fusion, not raw score comparisons across heterogeneous indices.
- Early, safe decisions: in low-risk cases, a verified A/B answer may be generated without C, while a mandatory-C case must wait or honestly report uncertainty. Never claim complete evidence after intentionally dropping an authoritative dependency.
- Transport optimization: reuse connections, avoid repeated authentication handshakes, cache safe metadata and honor source quotas; isolate pathological connector retries. Store the right document/version and tenant ACL in cache keys.
- Streaming and background workflows: streaming an initial token earlier can improve perceived response time but should not present an incomplete high-risk decision as final. For optional supplemental content, support clear post-answer follow-up only if product semantics permit it.
- Deadlines and cancellation: propagate absolute request deadlines, cancel unneeded optional branches, and avoid wasted work after client disconnect; Q279 goes deeper into bounded partial results and external API deadlines.
- Canary and rollback: ship source router, concurrency/fan-out limits, candidate budgets and sufficiency checks behind independent flags. Shadow-evaluate source coverage and p95 on multi-tenant samples, then canary without bypassing authorization/source currency.
7. Trade-offs, pitfalls and common mistakes
- More sources ≠ slower serial work: properly parallel source calls overlap, but the slowest required branch still determines barrier completion. More sources also raise probability of hitting a slow tail.
- Source routing: cuts unnecessary calls but can suppress rare, critical evidence. Use explicit high-risk routing overrides, query-level coverage audits and fallback on uncertainty.
- Early finish: can reduce tail latency but may return incomplete or misleading answers if C has mandatory facts. Ground the sufficiency test in a verified requirements inventory.
- Global candidate cap: improves compute but may disproportionately drop niche source evidence. Use diversified per-source/facet quotas and measure C-only recall.
- Cache: speeds repeated access but may leak cross-tenant content or serve expired policies when version/permission keys are weak. Verify authorization at retrieval/render time.
- Hedged duplicate requests: can reduce rare tails but raises downstream load and cost; never blindly hedge a rate-limited C. Use only under measured spare capacity and per-request budgets.
- Sequential source search: may avoid unnecessary work when A/B suffice, but makes true C-dependent requests slower; test task-conditioned routing.
- Reranker expansion: bringing 75 candidates into an expensive cross-encoder without dedup can consume more latency than the new retrieval source itself.
- Parallelism without isolation: C can exhaust the shared HTTP connector pool and worsen A/B; monitor queue wait per pool.
- Percentile math: adding stage p95s or averaging per-source p95s does not produce overall p95. Analyze timestamped per-request traces and critical-path dependencies.
- Prompt inflation: extra source text can increase generation prefill, monetary cost and hallucinations. Preserve sufficient authoritative context rather than blindly concatenate.
- Overoptimizing TTFT: a quick first token does not substitute for full answer freshness, completeness and citation support.
- False improvements: benchmark not only speed but source-specific answerability, unsupported citations and tail errors. A fast incorrect answer is a failed RAG request.
8. Prevention and production runbook
- Define a per-source contract: eligibility/routing rules, authorization, expected latency distribution, response size, rate limit, cost, source freshness, cache policy, failure behavior and mandatory-versus-optional semantics.
- Regression-test adding any new source using representative answerable-by-existing versus uniquely-answerable-by-new-source queries; measure p95/p99, TTFT, source contribution, candidate/LLM token inflation and wrong-source citations.
- Require distributed traces of every fan-out branch and downstream merge/rerank; alert on rising
fanout_width,C_call_fraction, source tail, shared-pool wait, cross-encoder candidate count and final prompt tokens. - Maintain golden queries requiring multiple sources and critical exceptions. A router that avoids C on ordinary questions but fails these cases must not pass rollout.
- Inject a delayed C dependency, retry storms, cache misses, connection-pool exhaustion, wrong source routing and oversized C candidate responses; verify failure isolation and safe fallbacks.
- For an incident: compare before/after paired traces → inspect fan-out dependencies → identify max branch/barrier → measure queue wait vs remote service → inspect candidate count and prompt tokens → run C-stub/fixed-candidate tests → test selective routing → recheck coverage and latency on the full cohort.
- Gate rollout on both latency (p95/p99, TTFT) and quality (C-only gold-span recall, evidence completeness, citation validity, no unauthorized leakage), as well as per-request token/connector cost.
- Keep source C feature flag and a conservative router rollback; fail closed for queries whose authoritative evidence cannot be fetched instead of silently substituting unrelated sources.
Interview takeaway: When a third RAG source doubles latency, find which dependencies are on the critical path, not just which span is slow. In parallel fan-out, the slowest required source controls the join barrier; candidate merge/rerank and extra prompt tokens add more delay. Use distributed traces and source-ablation/fixed-candidate experiments, then reduce unnecessary calls through query-aware routing, evidence sufficiency checks, bounded fan-out and candidate budgets while preserving unique-source recall.
Research references
- Dean and Barroso, *The Tail at Scale*, Communications of the ACM 2013: https://research.google/pubs/the-tail-at-scale/
- Google SRE, *Monitoring Distributed Systems—Tail Latency*: https://sre.google/sre-book/monitoring-distributed-systems/
- Microsoft Azure Architecture Center, *RAG Information Retrieval Phase*: https://learn.microsoft.com/en-us/azure/architecture/ai-ml/guide/rag/rag-information-retrieval
- Microsoft Azure AI Search, *RAG and AI Search*: https://learn.microsoft.com/en-us/azure/search/retrieval-augmented-generation-overview
- Microsoft Azure Architecture Center, *Agentic RAG Design*: https://learn.microsoft.com/en-us/azure/architecture/ai-ml/guide/rag/rag-agentic
Learning Sequence
Retrieval-Augmented Generation Curriculum
Explore Related Questions
Deepen your understanding across related Retrieval-Augmented Generation topics.