Retrieval-Augmented Generation
Retrieval-augmented generation connects a generative model to a searchable knowledge source. Instead of relying only on model weights, the system fetches relevant documents and places them into the prompt or context.
The model produces an answer using evidence from two different financial reporting periods. How would you enforce temporal consistency?
Production incident (illustrative): A finance RAG assistant receives “What was Helios Components' FY2026 operating margin?” It retrieves an annual report passage giving FY2026 revenue = ₹1,200 crore (April 2025–March 2026) and a near identical comparative table giving FY2025 operating profit = ₹180 crore (April 2024–March 2025). It then calculates 180/1,200 = 15% , cites both genuine passages and confidently labels the result “FY2026 operating margin.” The FY2026 operating profit, from a different page, was ₹144 crore: the correct FY2026 margin is 144/1,200 = 12% . The generated arithmetic is correct but the join between facts is invalid. Both snippets were retrieved, indexed and authorized; the failure is cross period evidence assembly and answer synthesis . Principle: A financial fact is not merely (metric, value, document). It must include reporting entity, accounting basis, period start/end or instant, fiscal calendar, unit, currency, segment/dimensions, filing/version and restatement status . Two true source facts are not necessarily composable. Enforce a typed temporal fact contract before any cross chunk aggregation, arithmetic, comparison or citation. SEC EDGAR's XBRL API and XBRL specifications represent reporting contexts explicitly; revenue and operating profit are duration facts, while many balance sheet amounts are instant facts. 1. Contain the production incident and define the intended period Do not return a mixed period KPI as a confident financial answer. If an exact period aligned numerator or denominator is missing, say what is unavailable or ask which period the user wants; never silently borrow a nearby fiscal year. Capture the user question; trusted reporting entity ID; financial reporting calendar and timezone; requested metric; canonical fiscal year/quarter; period start/period end; whether the question requests an as of balance , duration flow , trailing twelve month amount, growth comparison or restated figure. Trace retrieval through the final prompt: query, search filters, index generation, each chunk's source/version, document title, table row/column headers, extractable time context, model output, calculated formula and per number citations. Identify authoritative ground truth for FY2026: a canonical accounting data API or filing table with matching 2026 period, unit and consolidated/standalone scope. For a public issuer, SEC's companyfacts XBRL data can supply facts with date ranges and filing metadata; do not treat an XBRL calendar frame as a universal company fiscal quarter without date inspection. Respect tenant and document ACLs, retention, embargo and source effective dates. Diagnostic widening to an unauthorized filing is never a valid fix. 2. Separate the clocks that commonly get confused Reporting period: the actual interval for a flow fact, e.g. 2025 04 01..2026 03 31 for FY2026 revenue; or a single 2026 03 31 instant for closing cash. Fiscal period label: an entity specific FY2026, Q4 FY2026 or H1. The same fiscal label across companies does not guarantee identical dates. Some organizations use calendar years; many Indian companies close 31 March; always inspect dates. Publication/filing date: when a company published or filed the statement, possibly weeks after the period ended. A 2026 publication can contain FY2025 comparative values; a 2025 publication can reference forward looking 2026 forecasts. Data valid and system observed time: a restatement may be issued later but relate to an earlier reporting period. Record period end, filed at, ingested at, restatement version separately instead of using one ambiguous date field. Comparative column: an FY2026 annual report may have FY2026 and FY2025 side by side. Document level metadata fiscal year=2026 is insufficient to label every numeric cell FY2026. Instant vs duration: balance sheet values as of a date are not comparable by naive addition to period flows. XBRL describes instant and duration as distinct period types; validate start and end when applicable. Quarter to date / year to date: FY2026 Q2 YTD revenue covers April–September, while standalone Q2 revenue covers July–September. Do not mix period lengths, even if both are labelled “Q2.” Restated versus originally reported: restated prior period facts can be intentionally used when a user requests restated comparisons. Choose a documented basis consistently and label the version; avoid silently mixing original and restated numbers. 3. Root cause debugging sequence A. Query time interpretation: confirm the planner parsed “FY2026” into Helios' exact fiscal calendar, accounting basis and requested KPI. Check query rewrite and chat history carryover, especially “last year” relative to the company's reporting period rather than current wall clock. B. Source metadata: inspect whether every candidate numeric passage has a fact period and source version. Missing period start/period end is a provenance defect, not an invitation to assume document year. C. Table extraction: compare the original PDF/table with extracted cells and their column headers. OCR and markdown conversion can flatten “FY2026 / FY2025” into text with ambiguous numeric associations. Test header inheritance across pages and repeated table headings. D. Index granularity: see whether the chunk contains multiple comparative columns or includes a 2025 number under a 2026 document title. A one label per chunk metadata design cannot express mixed period facts safely. E. Retrieval filters: confirm the intended period filter is applied to a fact level index or correctly typed records. A parent document filter alone may admit both 2026 and 2025 figures. F. Selection/reranking: trace which candidate carried each of the two arithmetic inputs; check if the FY2026 numerator existed lower in rank but was dropped during rerank/packing. Rerank with matching metric and period, not only name and semantic similarity. G. Context assembly: inspect whether table headers, units, footnotes, currency and period labels survived compression. Repeated data blocks may be deduplicated incorrectly across years. H. Arithmetic reasoning: log parsed inputs as structured fact IDs. Recompute derived metrics outside free form generation. Reject a formula if operands have incompatible periods, inconsistent entity/consolidation scopes, mismatched units or currency. I. Output provenance: require each number in the answer to carry its own source fact ID and period. A citation to the annual report alone does not show that the cited row and fiscal column match the claim. J. Failure classification: source extraction mismatch, fact metadata missing, period filter error, wrong rerank choice, context header loss, unsupported cross period join, or answer/citation mismatch. Fix the first observed boundary rather than adjusting the LLM prompt blindly. 4. Typed fact model and compatibility guard Represent a fact as {entity id, metric id, value, unit, currency, period type: instant|duration, start date?, end date, fiscal calendar id, fiscal year, fiscal quarter?, period length days?, reporting scope, accounting standard, segment dimensions, source id, accession id?, filed at, restatement version, provenance span}. Prefer authoritative structured XBRL facts where available. For PDF only filings, extract candidate rows with inherited headers and validate against the source. Do not infer source periods from document names, embedding similarity, current dates or timestamps alone. Same period KPI guard: for a margin profit/revenue, both operands must be duration facts covering the same start and end dates and comparable accounting/entity scopes. Ensure revenue ≠ 0; preserve units, percentage scaling and rounding rules. As of guard: for a debt balance or cash balance, require instant facts as of the requested date, with applicable basis. Comparing cash at two dates is valid when expressly labelled as a change, not as a same period arithmetic input. Comparison guard: year on year growth intentionally uses two different comparable periods, but only under an explicit comparison plan: (FY2026 revenue − FY2025 revenue)/FY2025 revenue; validate corresponding fiscal durations and restatement policy. Cross period mixing is wrong when accidental , not when requested and labelled. Rolling period guard: for a trailing 12 month calculation, sum non overlapping, contiguous quarters or obtain a verified TTM fact; never sum YTD and full year values that overlap. Restatement guard: record whether prior year comparative is restated and whether current year values have been revised; choose authoritative current restated reporting consistently rather than assuming latest filing date equals latest period. Cross entity guard: an issuer's group report can include parent, subsidiary and consolidated columns. Bind entity id, reporting scope and segment dimensions before joining. This complements Q270's entity resolution work. Ambiguity guard: if two trusted sources disagree on period boundaries, definitions or restatements, surface an explicit conflict and request clarification or route to accounting review. 5. Beginner friendly numerical production example The FY2026 Helios annual report provides revenue ₹1,200 crore and operating profit ₹144 crore for 1 April 2025 to 31 March 2026 . The adjacent FY2025 comparative column provides revenue ₹1,000 crore and operating profit ₹180 crore for 1 April 2024 to 31 March 2025 . The broken RAG pipeline extracts FY2026 revenue 1200, FY2025 operating profit 180, and computes 180/1200 × 100 = 15% , incorrectly calling it FY2026 operating margin. A period consistency gate rejects this join because profit.period=[2024 04 01,2025 03 31] differs from revenue.period=[2025 04 01,2026 03 31]. The corrected join uses FY2026 facts: 144/1200 × 100 = 12% . For an explicit comparison, FY2025 operating margin was 180/1000 × 100 = 18% ; FY2026 margin is 6 percentage points lower , which is a legitimate cross period comparison because each margin was first calculated within its own consistent period. For a hypothetical 200 question evaluation set involving reported numbers, 160 questions have a single requested fiscal period and 40 require deliberate year over year comparisons. The baseline produces valid period aware, correctly supported answers for 126/200 = 63% . A fact level date/column extractor, typed fiscal context, compatibility checks and targeted refetch improve illustrative supported answer success to 178/200 = 89% on the same frozen, authorized queries, an absolute 26 percentage point increase . There are still 22 failures needing review of table parsing, restatement scope, source availability or generation. Suppose a further 12 requests with ambiguous fiscal/calendar mapping are held for clarification rather than answered; report answer rate , correct supported answer rate , period mismatch escape rate and abstention separately. The 178/200 figure is a controlled illustration rather than a claim about current MachinoAI traffic. A structured fact lookup could increase retrieval and validation p95 from 170 ms to 245 ms in this example. Accept the 75 ms overhead only if measured latency, cost, security and correctness gates meet service objectives. 6. Engineering solution and safe deployment Fact level indexing: index numeric facts or table cells with exact source row/column provenance plus normalized temporal metadata, rather than relying on one reporting year per PDF chunk. Keep the original excerpt and page/span for human verification. Authoritative fiscal calendar registry: maintain an effective dated calendar per reporting entity or business unit; store FY labels and actual start/end dates. Handle 52/53 week years, nonstandard reporting quarters and fiscal calendar changes explicitly. Source lineage: version facts by filing accession/document version and filed at. Retain restated facts and superseded values with explicit precedence instead of overwriting the reporting period itself. Typed retrieval planner: interpret query → resolve entity/metric/scope/period intent → form authorized fact search constraints → retrieve exact period aligned facts → fetch surrounding textual disclosure/explanations where needed. Two tier retrieval: use hybrid semantic/BM25 for descriptive narrative and section discovery, then authoritative structured lookup for numeric operands and their reporting contexts. Do not ask a semantic reranker to guarantee equality of financial periods. Validated calculation service: compute margin, growth, average or TTM with domain specific compatibility rules. Reject invalid period joins with a typed error such as PERIOD MISMATCH, INSTANT DURATION MIX, OVERLAPPING INTERVALS, INCOMPARABLE SCOPE or RESTATEMENT CONFLICT. Evidence manifest: attach fact id, entity, metric, period start/end, period type, currency, unit, scope, source version, and citation span to each input and calculated output. Narrative answer: state the fiscal period and context: “For FY2026 (1 Apr 2025–31 Mar 2026), operating margin was 12%, calculated from revenue ₹1,200 crore and operating profit ₹144 crore.” Cite each operand/source and label comparative figures explicitly. Uncertain evidence: when facts are missing, mixed, or unauthoritative, perform a bounded refetch with the correct period and source owner. If reconciliation fails, abstain or show the periods separately; never invent or silently backfill a numerator from another year. Rollout: shadow test against audited filings/structured facts, canary by fiscal calendar and document type, and gate on period alignment, correct calculations, version precedence, source citations, access controls, p95/p99 and cost. Preserve rollback that cannot resurrect revoked filings or unauthorized records. 7. Trade offs and common mistakes One document year metadata field: easy to implement but wrong for comparative tables, quarterly/YTD columns and restated numbers. Use fact level period attribution. Automatic “latest” selection: latest filing can contain a historical value. Ask what period the user means and inspect the XBRL fact's actual start/end or instant, not just filed at. Date filters alone: narrow the candidate corpus but cannot prove the chosen two values are compatible. Add post retrieval typed join validation. XBRL structured facts: high quality temporal contexts where available, but concepts, dimensions, units, fiscal calendars and restatement semantics still require interpretation; some narrative financial facts lack clean tags. PDF tables: broad coverage but OCR/table parsing can detach headers, units and comparative periods. Require source span verification and quality gates. Strict same period joins: essential for margins, but would incorrectly reject legitimate cross year growth and cash balance changes if the operation type is ignored. Model the permitted temporal relation per calculation. Period length equivalence: equal number of days does not guarantee the same fiscal quarter or comparable seasonal basis. Explicitly label and review nonstandard comparisons. Currency conversion: same fiscal dates but different currencies or units still need conversion rules and conversion date provenance; never assume two numbers share a unit. Restatement changes: a later filing may correct an older fiscal value; choosing “newest filed” indiscriminately can mix standards or scopes. Single tokenized chunk for whole table: can accidentally treat adjacent FY2025 and FY2026 cells as one number sequence. Separate rows/columns while retaining source references. LLM arithmetic: model generated formulas can be correct on wrong inputs. Typed operand validation and deterministic calculations provide a more auditable boundary. Latency: structured lookup and validation add I/O/CPU but can prevent expensive policy and investor facing errors; measure selective activation for factual numeric questions. 8. Prevention, monitoring and regression suite Build a temporal golden set across calendar years, FY labels, Q1–Q4, half years, YTD vs quarter standalone, TTM, closing balances, comparative columns, restatements and cross company fiscal calendars. Include explicit negative cases: FY2026 numerator + FY2025 denominator, 2026 publication containing 2025 comparative, Q2 YTD + Q2 standalone, 2026 consolidated profit + 2026 standalone revenue, and FY2025 original versus restated values. Alert on PERIOD MISMATCH, MISSING FACT PERIOD, COLUMN HEADER LOST, INSTANT DURATION MIX, UNVERIFIED RESTATEMENT and unsupported derived metrics; monitor by ingestion source and extraction version. Track period aligned fact recall , cross period join rejection rate, period mismatch escape rate , correctly supported calculation rate, period label citation correctness, abstention, latency and confidence intervals on labelled audit samples. Require every numeric extracted fact to retain a reporting context and exact source citation; reject index writes if a mandatory date or table column binding cannot be established, or quarantine the fact for manual review. Regression test on each document loader, OCR, chunking, table extractor, financial calendar mapping, query parser, model and prompt change. Verify both fact level contexts and full answer arithmetic. Preserve an audit record: original request, resolved temporal intent, candidate fact IDs, rejected joins and reasons, chosen operands, equation, output, versions and citation spans. Use source controlled, reviewed changes to fiscal calendars and precedence. A source reindex should not automatically make expired or differently dated facts eligible for an unrelated query. Interview takeaway: When a RAG answer merges two financial reporting periods, inspect the reporting period of every individual numerical fact , not just the file year or publication date. Make the join a typed operation that checks entity, period start/end, instant/duration, metric, scope, units and restatement context before calculation. Cross period comparisons are valid only when explicitly requested, individually period consistent and clearly labelled . Research references SEC EDGAR APIs and XBRL Company Facts: https://www.sec.gov/search filings/edgar application programming interfaces XBRL International, Dates in XBRL (instants and durations): https://www.xbrl.org/dates in xbrl/ FASB, Period Type Selection guidance: https://xbrl.fasb.org/resources/Style Guides/periodtypeselection.pdf Microsoft Azure AI Search, Text Query Filters : https://learn.microsoft.com/en us/azure/search/search filters Microsoft Azure Architecture Center, RAG Information Retrieval : https://learn.microsoft.com/en us/azure/architecture/ai ml/guide/rag/rag information retrieval
A RAG assistant gives different answers to the same question across repeated requests. How would you separate retrieval nondeterminism from generation nondeterminism?
Production incident (illustrative): A procurement RAG assistant answers the same question—“Who must authorize Singapore contractor lodging exceptions in 2026?”—three ways across five requests: “Regional Finance Director,” “line manager,” and “finance team.” All requests return HTTP 200. One trace cites the current contractor exception policy; another cites an older employee travel FAQ; a third cites the correct policy but paraphrases its approval condition incorrectly. Changing temperature to zero does not eliminate all differences. The team must identify whether the evidence changed, the final model input changed, or the model interpreted identical evidence differently. Core principle: Diagnose repeat answer variability with a three boundary replay : (1) hold the natural language request fixed and inspect retrieval; (2) hold the retrieved candidates fixed and inspect reranking/context assembly; (3) hold the exact serialized model input fixed and inspect generation. Separate lexical variation (different wording, same supported decision) from semantic/grounding instability (different approvals, amounts, dates, exceptions or citations). Zero temperature and a seed can improve repeatability but are not absolute guarantees across all model/providers and deployments. 1. Contain the incident and define “same request” Verify exact user question, conversation history, user/tenant identity, ACL, locale/timezone, query timestamp, feature flags, model routing, source snapshot, and effective date context. A request containing “today,” an updated chat memory, or changed user permission is not the same semantic task even if the last user message is identical. For decisions involving approval authority, policy eligibility or money, avoid selecting the most frequent answer by majority vote if cited evidence conflicts. Require current authorized source support; abstain or clarify when the controlling rule is unavailable. Capture request and stage correlation IDs: raw input hash, normalized/rewrite text, query vector hash, embedding model/revision, source alias/index generation, filter JSON, ANN parameters, selected shard/replica, dense/BM25 candidates with ranks, fusion/rerank output, packed context, final serialized prompt hash, model name/revision, sampling settings, seed if supported, response fingerprint if supplied, output text and citations. Keep trace payloads access controlled; hashes and structured candidate IDs often suffice for production telemetry. Securely retain exact text only for approved diagnostic replay. The test harness must respect source ACLs and deletion/retention policy. Label the correct answer from a current authoritative passage with source/version/offset. Set task specific invariants: correct approver, approval before booking, applicable jurisdiction, no employee policy substitution and properly supported citations. 2. Classify variability before debugging Style only: “Director of Regional Finance” vs “Regional Finance Director” can be equivalent if both cite the same valid clause; exact string mismatch is not necessarily a defect. Evidence variation: candidate IDs, content hashes, source versions or scope filters change. First investigate retrieval, reranking, context assembly, source freshness or access control. Generation variation: the exact same serialized prompt and model settings yield different decisions, unsupported claims, omission of exceptions, or citations. Inspect decoding/model versions and grounding behavior. Hybrid/multistage variance: retrieval is stable, but dynamic query routing, LLM query rewriting, reranking or packing changes the prompt. This is neither first stage vector instability nor pure decoder randomness. Temporal/version change: documents legitimately update during the test. Different answers can be correct for different as of timestamps; compare only when the authorized source snapshot and requested effective date are controlled. Failure severity: a changed adjective is less severe than a changed numeric threshold, policy approver, effective date or unauthorized citation. Measure critical decision consistency separately from text similarity. 3. The four cell controlled replay matrix A. End to end repeated requests: issue 20–50 identical, authorized test requests under a pinned session/tenant/source snapshot. Record all stage outputs. This measures observed variation but does not yet attribute cause. B. Frozen input + retrieval only: generate the embedding once, pin the literal query vector, filters, index alias/generation, ANN settings and ideally a replica. Run vector and lexical search repeatedly without calling the answer model. Compare top k source IDs, content hashes, authorized relevant spans and exact/ANN reference results. C. Frozen retrieval + assembly only: inject a fixed, ordered list of authorized candidate objects and deterministic reranker inputs into the context builder; rerun fusion, deduplication, compression, sorting, token packing and citation serialization. Compare exact final prompt hashes and included source offsets. If these change while input candidates are fixed, context construction is unstable. D. Frozen exact prompt + generation only: replay precisely the same serialized system/developer/user messages, tool outputs, evidence spans and model settings directly to the generator repeatedly. Compare answer decisions and citations with a domain aware evaluator. Changing output with an identical prompt indicates generation/backend nondeterminism, not vector retrieval. E. Cross factor intervention: replay multiple fixed candidate lists with fixed prompt templates across matched model revisions. Use randomized, balanced assignments if evaluating whether each retrieval variant causes a response change; do not compare sequential samples after an unrecorded model release. F. Hold timestamps and caches constant: pin source content and policy effective date for a diagnostic run. Record cache keys; two requests with the same query text but different selected source generations or cache entries are not equal inputs. G. Independent reference: if ANN results vary, compare exact/flat nearest neighbors on the same filtered corpus ; if the final prompt varies, compare its token/byte level diff; if generation varies, check model/provider fingerprints, sampling and actual response content. 4. Retrieval side root cause debugging Embedding/preprocessing: capture query rewriting, spelling normalization, chat history condensation, query embedding inputs and the resulting vector fingerprint. An LLM rewriter can produce different search queries even when the user question is unchanged; pin its result when testing search itself. Approximate search: HNSW and other ANN methods can change the rank order of nearby candidates, especially around top k boundaries. Qdrant documents slight ranking shifts and recommends exact search when stable ordering is required. Determine whether the changed neighbors affect actual evidence coverage . Replica consistency: a current policy can be visible on one replica and not another during updates. Qdrant documents configurable read consistency and read affinity; investigate peer specific source versions, optimizer readiness and deletion propagation. Affinity may hide rather than repair stale state. Filters/time dependent policies: verify tenant, role, region, status, source version and effective date are serialized identically. Avoid relaxing ACLs to make results consistent. A current and an expired source are not interchangeable. Fusion and reranking: compare branch depths, RRF/rank fusion, score normalization, candidate deduplication, stochastic LLM rerankers and tie breaking. If first stage candidates match but their final order changes, label the issue “selection/assembly,” not vector retrieval. Concurrency: production load, partial shard responses, timeouts, retries, caches and background ingestion can change candidate completeness. Distinguish a successful complete response from a partial result treated as success. Source changes: pin document/chunk IDs, content/version hashes and index alias. Identical looking citations may point to altered content if the resolver follows a mutable latest version URL. 5. Generation side root cause debugging Exact prompt equality: assert stable serialized messages, source spans, instruction order, context section order and tool content. A stable “question” field alone does not prove identical model input. Sampling parameters: record temperature (if supported), top p/top k, max output tokens, stop sequences and seed. Provider support differs by model; OpenAI documents best effort reproducibility with fixed seed plus system fingerprint on applicable APIs, while even temperature zero is not a universal determinism promise. Backend/model drift: compare immutable model snapshot where available, provider version, system fingerprint if provided, inference region, safety settings and tool use mode. A model alias can move to a new revision even when client config stays the same. Evidence interpretation: same current policy can generate different approval outcomes because of negation, exception omission, ambiguous pronouns, prompt length, position sensitivity or conflicting older policies. Evaluate claim support and qualified decision correctness , not only string consistency. Output formatting/citations: test whether source IDs are validated by a deterministic resolver or generated as free text. Different URLs/citation labels can be a citation rendering defect, separate from decision content. Length and termination: output token caps or stop conditions may truncate an exception in only some runs, causing apparent semantic disagreement. Record finish reason and generation tokens. Independent reviewer: judge semantic equivalence and citation support against a verified source. An LLM as judge has its own error rate; human audit critical disagreements. 6. Worked numerical production example Take 100 distinct expert labelled queries , each run 10 times (1,000 responses) under the same authorized policy snapshot. Define a stable decision as the same correct, source supported approval outcome across all ten attempts, not identical wording. Baseline: for 30 queries , the retrieved authoritative source set changes across repetitions; 20 of these exhibit a changed final decision. For 70 queries , retrieval and final prompt stay byte identical; 18 still exhibit changed decisions. The other 62 queries remain decision stable. Thus 38/100 = 38% of query groups show material answer variability, but one cannot assign all 38 to retrieval: 20 are retrieval associated; 18 occur with fixed prompts and demonstrate generation side variation. One affected query has source lists R1=[CurrentPolicy, ExceptionMemo, FAQ] and R2=[OldEmployeeFAQ, CurrentPolicy, FAQ]. Its citation set Jaccard is 2/4=0.5 when distinct source IDs are used. The correct authoritative exception survives in R1 but not in the final packed prompt for R2. This path is retrieval/selection associated, not proof that ANN itself failed. Controlled retrieval only replay pins the vector and filter, revealing a source version mismatch on two replicas. Correcting replica freshness and stable source routing resolves 16 of the 20 retrieval associated variable decisions, leaving 4 whose candidate/packing changes require further investigation. Frozen exact prompt generation replay reproduces decision differences for 18 query groups. Fix an ambiguous instruction, add span grounded approval verification and use supported low variance decoding settings. Suppose this eliminates variability for 14 of 18 , leaving 4 generation sensitive groups for further human review. On the same 100 query benchmark, assume there is no overlap between the 20 retrieval associated and 18 frozen prompt groups as defined in this controlled example. The remaining variable groups total 4+4=8/100 = 8% , down from 38% , a 30 percentage point reduction . In real deployments, interactions/overlap mean you must remeasure end to end variance rather than naively add reductions. Suppose p95 rises from 150 ms to 205 ms due to stronger consistency, extra provenance checks and verification. This is +55 ms . These numbers are illustrative ; measure latency, freshness, availability, grounded accuracy and cost before rollout. An unchanged answer that is consistently wrong is not success . The evaluator must also check accuracy and citations: deterministically retrieving an expired employee policy would reduce variance while increasing harm. 7. Engineering solution and safe rollout Versioned trace contract: correlate query hash, rewritten query hash, embedding hash, retrieval manifest hash, packed prompt hash, answer claim hash, source versions, ACL snapshot, planner/model revisions and replica identity across requests. Stage specific reproducibility tests: on every release, rerun (i) fixed vector retrieval, (ii) fixed candidate context packing, and (iii) frozen prompt generation. Each should report variation independent of other stages. Canonicalize for metrics: use stable source/version/section IDs to compare retrieval sets; for answers, compare structured decision fields (approver, condition, amount, date), normalized claims and citation support instead of punctuation or stylistic text. Reduce source variance: pin embedding and index generations, inspect replica consistency, benchmark ANN candidate depth against exact search, and detect missing/deleted source versions. Stronger consistency has latency and throughput costs. Stabilize selection: deterministic sorting with documented tie breaks where applicable, stable fusion windows, version aware deduplication and a consistent context budget. Where LLM query rewrite/rerank is used, capture and reuse its outputs for replay. Improve grounded generation: require current authorized spans for each decision, preserve exception qualifications, verify claims after generation and abstain when evidence is ambiguous. Low temperature alone does not fix missing or conflicting evidence. Provider aware reproducibility: on an API supporting seeds, hold seed and all settings fixed and record backend fingerprints. Treat exact output matching as best effort; benchmark on task correctness and citations, not a promise of byte for byte generation. Shadow then canary: roll out corrected retrieval settings and generation controls independently where possible. Attribute changes to each intervention using controlled cohorts, and monitor variance and accuracy together. Safeguarded rollback: roll back config/model versions without reviving deleted or unauthorized documents. Retain current ACL and policy validity checks regardless of the candidate source index. 8. Trade offs, common mistakes and limitations Temperature zero: usually reduces sampling randomness when supported; it does not freeze prompt assembly, backend model revisions, GPU execution or ambiguous reasoning. Fixed seeds: useful in supported APIs but provider/model dependent and may still differ after system changes; never claim guaranteed determinism. Exact search: provides a more stable diagnostic reference than ANN but costs more latency/compute for large corpora. Use selective exact checks for critical query slices. Read consistency/affinity: can reduce replica dependent blinking, but stronger reads add load and fixed affinity can conceal a lagging replica. Verify deletion/ACL propagation. Caching responses: makes repeated strings identical but can lock in wrong answers, stale documents or permissions; key and invalidate by tenant, source version, ACL policy and effective date. Majority vote across generations: can produce a confident consistently wrong answer if all generations interpret one stale policy incorrectly. Prefer verifiable authoritative evidence over consensus. Prompt simplification: reduces some variation but may remove qualifiers and increase omission errors; preserve source and citation conditions. Repeated run cost: 10–50 repeats per query can be expensive; use a stratified golden set, adaptive sampling and human review for high impact disagreements. Evaluation granularity: overlap@k and embedding similarity can show candidate movement but do not prove harmful answer drift. Measure actual decision invariants and supported citations. Stochastic infrastructure: retries, fallbacks, load balancing and indexed data freshness can change the complete retrieval graph; trace them instead of assuming “same API call = same inputs.” 9. Prevention checklist and production runbook Maintain a golden set with policy exceptions, current versus expired documents, equivalent phrasings, fixed number approvals, ambiguous citations and cases whose relevant source is near the top k boundary. Run fixed query repeated end to end , fixed vector retrieval only , fixed candidate assembly only and frozen prompt generation only suites after changes to model, indexing, rerank, prompt, tokenizer, deployment or source routing. Alert separately on retrieval candidate variance , final prompt hash variance , material decision variance , unsupported claim rate, wrong policy citation rate, stale replica reads and p95/p99. Log exact source/version and effective date applicability for every decision; check ACL and current deletion status before answering even when retrieval is cached. Use human adjudication for semantic disagreements; measure pairwise both correct rate, repeated run all correct decision rate, and proportion of responses with consistent but incorrect answers. Inject faults: one stale replica, inconsistent index alias, changed query rewriter output, unstable fusion tie break, dynamic chat history, model rollout and a changed provider fingerprint. Set clear rollback triggers based on grounded decision correctness , not merely response consistency. A stable hallucination is still a failure. Interview takeaway: Repeated RAG answers can vary because retrieval returned different evidence, context assembly produced different prompts, or generation changed despite identical prompts . Freeze and replay each boundary independently— fixed vector, fixed candidate list, and exact serialized prompt —then measure material decision and citation consistency alongside factual correctness. Adjust the first failing stage rather than blindly setting temperature to zero. Research references OpenAI, Advanced usage: reproducible outputs, seed and system fingerprint: https://developers.openai.com/api/docs/guides/advanced usage OpenAI Cookbook, Reproducible outputs with seed: https://developers.openai.com/cookbook/examples/reproducible outputs with the seed parameter Qdrant, Search and ranking stability FAQ: https://qdrant.tech/documentation/faq/qdrant fundamentals/ Qdrant, Consistency guarantees and read affinity: https://qdrant.tech/documentation/scaling/consistency guarantees/ Redis, RAG debugging and failure stage diagnosis (2026): https://redis.io/blog/rag debugging guide retrieval errors/
Users report that answers are accurate but consistently omit important details. How would you measure and debug completeness?
Production incident (illustrative): A procurement RAG assistant answers, “What must our team do before booking a Singapore contractor's hotel above the SGD 350 limit?” It correctly says “obtain written Regional Finance Director approval” , and cites the governing policy. But it routinely omits three material details: an accessibility or documented business justification, approval before booking , and storing the approval with the reimbursement claim. Users report that the answers are accurate but not actionable. The underlying clauses exist in the authorized corpus. The incident is systematic answer incompleteness , not necessarily a hallucination. Core distinction: Groundedness/faithfulness asks whether the information stated is supported by source evidence (precision like). Completeness asks whether the response covers all material, applicable requirements of the question and trusted ground truth (recall like). A short, true answer can fail the user's task. Microsoft Foundry distinguishes groundedness from response completeness, and Amazon Bedrock evaluates how well an answer resolves all aspects of a question. Measure both, plus verbosity and citation validity. 1. Contain the incident and define what a complete answer means Establish exact user intent, permissions, domain and risk. “What is the cap?” may need a single number; “How do I request an exception?” needs eligibility, approval authority, timing, documentation and next steps. Do not demand unrelated facts simply to maximize coverage. Acquire an authoritative, current, permission appropriate answer specification from source owners, including mandatory and optional facts. For each expected fact, record a minimal supporting span and applicability conditions. Separate material omissions from stylistic brevity. Trace the specific request through source snapshot, query planner, source selection, retrieved candidates, reranker, dedup/compression, final model visible context, structured reasoning and final response/citations. Retain source IDs, versions, user role, offsets and a secure prompt manifest. Hold unsupported or incomplete high impact instructions for bounded refetch, targeted clarification or escalation. A partial finance policy answer should not be presented as the full procedure when omitted prerequisites could change the decision. Diagnose overlapping prior lessons precisely: Q266 covers retrieved sentence lost during trimming , Q267 covers position sensitivity despite prompt presence , Q268 covers unsupported stated claims , Q269 covers a specifically omitted conditional exception , and Q275 will isolate coverage for multipart questions . Q273 establishes general answer completeness measurement and stage by stage root cause across facts, workflows and question types. 2. Build an answer fact coverage contract Represent expected content as structured atomic answer units : requirement id, question intent, fact or step, required if, importance weight, source id, version, source span, authority, answer support criteria. Example booking exception checklist: F1: default SGD350/night cap; F2: qualifying accessibility/business case; F3: Regional Finance Director written approval; F4: approval obtained BEFORE booking; F5: retain approval with expense claim. Assign weights reflecting consequences, not word count: F1=1, F2=2, F3=3, F4=3, F5=1. A missing required approver or deadline should count more than a missing introductory definition. Define applicability : a case requiring approval has F2–F5; a user asking only for the ordinary cap may not. Avoid penalizing answers for excluding facts outside the user's scope. Include dependency relationships and acceptable paraphrases. “Written sign off before booking” may cover approval medium/timing when tied to the correct authority, but “finance must review” does not prove the Regional Finance Director requirement. Record authority/time/region/role. A fact about an obsolete employee policy is not “covered” for a 2026 contractor question, even if it looks similar. Allow “cannot determine from authorized evidence” as a calibrated uncertainty outcome , never as equivalent to a fully covered answer. Track answerability and abstention separately. Human adjudicators should approve gold fact sets; LLM extracted coverage units are useful candidates, not authoritative truth. Keep disagreements and changes versioned. 3. Evaluate completeness at fact, answer and task levels Atomic fact coverage: count expected applicable fact units correctly expressed and supported; fact recall = covered required facts / total applicable required facts. Weighted material completeness: sum(weights of correctly expressed mandatory facts) / sum(weights of applicable mandatory facts). In the example, the assistant mentions F1 and F3 only: (1+3)/(1+2+3+3+1)=4/10=40%; mere sentence level accuracy can still be high. All critical facts present rate: a binary per question gate requiring every high severity unit, regardless of weighted average. An answer missing “before booking” may be an unsafe instruction even if its remaining coverage is 90%. End to end task resolution rate: domain reviewers confirm that the response gives the required procedure or decision without invented steps. Distinguish completeness from correctness and user satisfaction. Unsupported claim rate / citation precision: measure alongside fact recall; adding generic or fabricated content can raise length and apparent coverage without improving grounded completeness. Coverage precision / verbosity: label unnecessary, irrelevant or repeated facts; an answer can be complete yet unwieldy. Report useful facts per 100 output tokens and response latency. Per stage evidence coverage: what fraction of gold fact units is in the source, retrieved candidates, reranked set, final prompt and generated answer? This localizes the first lost fact. Missing fact severity and slice metrics: report by domain, source/section, question length, user intent, policy exceptions, tables, language, factual/procedural/comparative questions, and token budget. A high overall metric can hide repeated omissions in one critical workflow. Evaluator reliability: use normalized propositions with subject/time/condition bindings and source linked passages; human score a calibration set to compute detector precision/recall and false positives. One LLM judge is not a perfect arbiter of “important.” 4. Root cause debugging: localize the missing information A. Gold/source gap: if a required fact is missing or outdated in the knowledge base, route to source owners; retrieval cannot produce evidence absent from the authorized corpus. B. Planning gap: did the system parse “what do I need to do” as one fact (“who approves”) instead of a full process? Compare the intended sub goal checklist against planned queries. For general completeness, ensure workflow steps are represented before retrieval. C. Retrieval gap: if gold spans exist in the index but are absent from candidates, compare per fact Recall@k, dense/BM25 branches, filters, query rewriting and parent section expansion. Increasing top k alone may add redundant generic passages without the missing footnote. D. Rerank/selection gap: if F4 was retrieved but pruned during reranking or source deduplication, inspect score versus novelty, group by source, candidate budgets and rank window decisions. E. Packing/compression gap: if the final prompt lacks the approval timing or document retention sentence, compare chunk hashes, source offsets, token budgets and dropped span reason codes (Q266). Preserve relevant adjacent sentences and headings. F. Reading/attention gap: if all required facts appear in the prompt but those in the middle are overlooked, test controlled position swaps and context reduction (Q267). G. Generation gap: if the prompt clearly contains all facts and the answer omits F2, F4 and F5, inspect “answer briefly” instructions, response token caps, template constraints, under specified completion criteria and miscalibrated summarization. Compare a structured fact ledger generation with the current free form response. H. Post processing gap: a streaming cutoff, markdown formatter, answer summarizer, UI character cap or citation renderer may remove sections after generation. Compare raw model output to displayed response. I. Evaluation gap: a metric measuring only answer relevance or claim groundedness may label the short response good. Inspect evaluator prompts, reference labels, weighting and human disagreement. J. Ambiguity gap: if the question omits facts necessary to determine requirements, the system should surface the missing attribute or ask one targeted question—not guess or claim full completeness. 5. Beginner friendly numerical production example Create 200 authorized, expert labelled enterprise questions . Each has exactly 5 applicable required answer units , for a total of 1,000 mandatory fact opportunities . This is a controlled illustration, not MachinoAI production data. Suppose the current system includes and supports 700/1,000 = 70% required units, even though 190/200 = 95% of its stated claims are judged grounded and mostly accurate. Distinct denominators explain why accuracy/faithfulness scores can mask omissions. Stage wise, the source corpus has 1,000/1,000 units; retrieval finds 940/1,000 = 94% ; reranking retains 900/1,000 = 90% ; final context contains 820/1,000 = 82% ; the generator expresses only 700/1,000 = 70% . Lost units are: retrieval 60 , rerank 40 , packing 80 , generation 120 . These stage losses are disjoint by first loss in the illustrative funnel and total 300 . The team repairs the dominant first loss stages: coverage aware retrieval/selection, span preserving context packing, and structured required fact checking. Suppose retrieval reaches 970 , rerank 950 , context 930 , output 900 required facts out of 1,000, for 90% fact recall —a 20 percentage point gain . On the same 200 question set, suppose all required facts answer success improves from 68/200 = 34% to 142/200 = 71% . This is not derivable from aggregate fact recall; it is measured separately on question level labels. For the five unit hotel example, the original answer contains F1+F3 (weighted 4/10 = 40% ); the repaired answer contains all five (weighted 10/10 = 100% ) while preserving accurate citations and conditional applicability. Added evidence checks increase illustrative p95 latency from 185 ms to 275 ms , a 90 ms overhead, and output tokens from 110 to 155 on average. Track costs and critical fact omission rate , not only raw coverage; a one line correct answer may remain preferable for a one fact question. Do not game the metric by producing longer lists: a generated “fact” counts only when applicable, correct, supported and substantively responsive to the requested task. 6. Engineering solution: fact coverage aware answer assembly Plan expected outputs: classify question type and derive a small scoped answer checklist (steps, prerequisites, constraints, actors, quantitative facts, timelines or comparisons). Ground the checklist in authoritative source labels rather than generating arbitrary “nice to have” details. Retrieve for coverage: use subqueries or controlled section expansion for distinct checklist items; compare required fact coverage of candidate sets, not only top k relevance. A fact aware selector can prefer one relevant exception paragraph to five duplicate summaries. Preserve provenance: attach stable fact id, document version, source span, question requirement id to every candidate; carry them through rerank, compression, packing, answer and citations. Budget by important facts: allocate context for high priority, nonredundant evidence; avoid trimming the only sentence supporting an important prerequisite. Use the real model tokenizer and verify the final serialized request. Generate from a coverage ledger: construct a structured internal mapping required fact → supporting span → covered/unknown before writing natural language. Then render a concise user response with clear steps and citations; do not expose irrelevant ledger detail. Pre release completeness gate: compare generated answer against the validated required fact ledger using deterministic checks for amounts, names, dates, enums and an entailment/LLM judge for paraphrases. Block critical omissions, allow controlled refetch/rewrite, and reverify. Separate omission from hallucination: a verifier should grade both precision of stated claims and recall of required facts , plus question level all critical completion. A correct but incomplete answer should trigger a targeted addition, not an indiscriminate rewrite. Risk aware escalation: if a critical fact remains unavailable after bounded retries, state the limitation or ask a clarification rather than improvising. Enforce permissions during all source expansion and repair loops. Feature flag rollout: evaluate against a held out gold set and shadow traffic, then canary by domain with audit samples for fact recall, all critical coverage, wrong/unsupported facts, output verbosity, p95/p99 and user task success. 7. Trade offs, failure scenarios and common mistakes Longer answers can contain more facts but also add noise, unsupported assertions and tokens; optimize relevant supported facts per answer , not length. Generic completeness prompts (“include all details”) rarely define what is material. Use task specific, authoritative expected fact specifications. Reference free grading is flexible for unlabelled traffic but cannot reliably distinguish missing facts from facts that were never applicable. Calibrate against human labelled gold answers. Aggressive subquery expansion increases recall but also latency, cost and opportunities for contradictory policies; require a bounded, authorized source set. One chunk per document can save tokens but delete separate required sections. Prefer span aware selection and document linked neighboring paragraphs. Fact weights reflect business risk but can be subjective; owners should approve them, and reports should show weighted and unweighted recall rather than hiding severe omissions in one aggregate. Coverage judge errors include false claims of support due to overlapping keywords, paraphrase rejection and missed negation; test numeric and modality perturbations. Question ambiguity may require clarification; do not label a justified uncertainty as an omission when the source genuinely cannot determine the answer. Overfitting to reference text can penalize valid shorter paraphrases; compare semantic claim coverage with stable entity, time and conditional scope. Source authority conflicts can yield incompatible facts; the correct response may need to state the disagreement instead of “covering” both as simultaneously valid. UI truncation may hide output sections even if the generator produced them; check the rendered user experience. Stable partial answers may look consistent in repeated runs (Q272) yet remain incomplete. Consistency and completeness are independent dimensions. 8. Monitoring and prevention Maintain a gold requirement checklist for representative workflows and critical policy decisions, with source/version offsets, expected units, applicability and reviewer approved weights. Capture stage level required fact id survival: source present → retrievable → reranked → packed → generated → displayed. Log first loss reason categories and compare with independently audited samples. Dashboard unweighted and weighted fact recall , all critical facts present rate , end to end task resolution, unsupported claim rate, citation precision, verbosity, abstention and p95/p99 by domain and question type. Build regression cases for missing prerequisites, dates, units, approval timelines, multi step processes, cross document instructions, tables/footnotes, negation and hard token caps. Inject controlled failures: remove a required paragraph from retrieval, clip a context chunk, insert repeated summaries, set low output max tokens, and truncate the UI; assert that alarms and safe fallback trigger. Re run coverage checks after policy changes, source owner updates, indexing/chunking, reranking, context compression, prompt template changes and model upgrades. Previously complete answers can become outdated. Human review high impact, highly confident but incomplete responses to catch issues automated judges miss; add these failures to the benchmark. Guard against data leakage by evaluating only evidence permitted to the requester; missing unauthorized details must not be counted as completeness failures. Define deployment gates on critical fact omission escapes , grounded accuracy and answer usefulness, not a single aggregate judge score. Interview takeaway: “Accurate but missing important details” is a recall/completeness failure, not necessarily a hallucination. Define the applicable required fact set , measure its survival from corpus through retrieval, reranking, packing, generation and display, and repair the first stage that loses each fact . Use both groundedness (stated fact precision) and completeness (required fact recall) , with critical fact gates and verified citations. Research references Microsoft Foundry, RAG Evaluators—Groundedness and Response Completeness : https://learn.microsoft.com/en us/azure/ai foundry/concepts/evaluation evaluators/rag evaluators Microsoft Azure Architecture Center, RAG LLM End to End Evaluation Phase : https://learn.microsoft.com/en us/azure/architecture/ai ml/guide/rag/rag llm evaluation phase Amazon Bedrock, Use metrics to understand RAG system performance : https://docs.aws.amazon.com/bedrock/latest/userguide/knowledge base evaluation metrics.html Amazon Bedrock, Retrieving information from data sources : https://docs.aws.amazon.com/bedrock/latest/userguide/kb how retrieval.html
A model starts producing invented source URLs despite receiving valid document IDs. How would you make citation generation deterministic and verifiable?
Production incident (illustrative): An internal procurement RAG assistant correctly retrieves doc id=policy sg 2026 v3 and a source excerpt stating that Singapore contractor travel exceptions require written Regional Finance Director authorization. It generates a useful answer but cites https://policies.example.com/sg/contractor expense 2026—a URL that was never returned by retrieval or stored in the document registry . In production, a similarly plausible path happens to resolve to an unrelated outdated page. Users trust the clickable link, despite the citation being fabricated. The correct document identity existed throughout retrieval; the defect occurred when the model was permitted to invent the presentation URL . Core design rule: A model can decide which allowlisted source reference IDs support a claim, but it must never be the authority that invents or resolves a citation URL. Build a trusted, deterministic reference registry → validated citation ID → authorized document/span resolver → link renderer . Distinguish citation existence , source authorization , URL integrity , source version , actual claim support , and link reachability . A link that resolves successfully can still be the wrong citation. 1. Contain the incident and establish the ground truth Stop rendering free form model generated links as verified citations. If a citation token cannot be resolved to a source in the exact retrieval response, suppress the link, block high risk answers or give a transparent “source link unavailable” state. Collect the source retrieval manifest for the affected request: request id, tenant/principal, source/document IDs, chunk/section offsets, source version, title, canonical source URI or approved lookup handle, content hash, connector ID, permission scope, and source effective date. Compare three artifacts: retriever references , the exact serialized model visible source IDs , and the final answer's citation IDs/URLs . Determine the first stage where an unsupported URL appears. An LLM generated citation URL can be fabricated even when document metadata and answer text are correct. Revalidate source policy: the user must have access to the underlying record and the intended version; a link must not allow cross tenant document access or reveal protected source identifiers. Never “repair” a fabricated citation by searching the open web for the closest looking URL: doing so can map an unsupported claim to the wrong document or expose private titles in search queries. Record whether the source has a stable external canonical URL, a connector specific document handle, an ephemeral signed URL, or no user accessible link at all. Not every document ID should be converted to a public URL. 2. Root cause debugging: locate the citation failure A. Source ingestion: inspect whether a source URI exists, is immutable/canonical or is an unstable file path, and whether the document moved or was versioned. A document ID can remain stable when its original storage location changes. B. Retrieval result schema: check whether the retriever returns doc id, chunk id, source id, title and source uri separately. A display title or relative storage path is not a safe source URL. C. Reference mapping: confirm that result local citation numbers such as [1] map unambiguously to the original returned reference id, rather than list positions after reranking, deduplication or context trimming. D. Context assembly: ensure every included evidence span receives a stable request scoped citation token mapped to its document/version/span. If a chunk is excluded from the final prompt, its token must not remain available for an unsupported generated citation. E. Generation instructions: examine whether a prompt asks the LLM to “include full source URLs.” That asks the model to synthesize information it cannot verify. Replace with an instruction to emit citation reference IDs only from the supplied enumeration. F. Decoding/schema validation: ensure free form markdown links or model produced url fields are rejected. A JSON schema or tool output contract can constrain format , but a malicious or mistaken permitted looking URL still requires trusted registry lookup. G. Post processing: inspect regex replacements, URL auto linkers, markdown renderers and citation index shifting. A formatter can turn a harmless source title into a fabricated URL or associate the right numbered marker with the wrong record. H. Redirect behavior: check if a legitimate canonical link now redirects to another tenant, domain, latest version record, login screen or outdated content. Validate each hop and final provenance; HTTP 200 alone does not prove correctness. I. Cached references: stale cached answers or citation maps can survive document deletion, role changes and file moves. Cache by source generation, tenant and access context; invalidate on ACL, version and URI changes. J. Answer grounding: a link to the right file may still attach to a paragraph that does not support the claim. Source resolution is distinct from Q268's claim entailment verification. 3. Trusted citation architecture Ingestion registry: maintain an authoritative source id, stable document id, document version, chunk id, optional page and span offsets, canonical connector handle, approved source URL when available, content hash, source owner, tenant and ACL. Request local evidence set: after retrieval and packing, create ref A, ref B etc. Each points to a verified model visible span and immutable evidence metadata. The LLM sees only this allowlisted reference set. Constrained output: return structured claims with text and citation ref ids: ["ref A"]. Accept only references in the current request's allowed set; avoid asking for url, hostname, or path in the model output. Server side resolution: a trusted service looks up ref A → tenant/source/document/version/span, checks authorization and freshness, then obtains an approved URL or an authenticated document viewer route. The model is not asked to create paths. Rendering: the UI converts validated IDs into clickable citations using only the resolver's returned destination. If no safe link exists, render a citation title/source label or controlled source preview without a fabricated href. Citation granularity: attach citations to the smallest factual assertion where practical, with page/section/offset when supported. Store exact source text hashes to prevent shifts after document changes. Document moves and versioning: stable document identity must survive path changes; the registry maps an identity to the current approved location while preserving citations to applicable historical versions when legally allowed. Deterministic mapping: sort or assign reference IDs consistently in the frozen evidence manifest. Do not rely on the LLM's token order for URL selection; identical manifests plus authorization state should render the same links. Permissions: issue short lived signed document links only after each request's authorization; never embed bearer tokens or long lived signed URLs in model prompts or caches. A signed URL may expire without invalidating the underlying citation identity. 4. Worked numerical example In an illustrative benchmark, 200 RAG answers contain 600 citation mentions across 120 unique documents. The generation model receives all 120 correct document IDs, but free form URL generation invents or corrupts 54 of 600 links : 9% invented link rate . Of 54 invalid links, assume 30 return 404 , 14 resolve to a different document , and 10 resemble valid links but point to outdated versions . Checking HTTP status alone catches at most 30, missing the 24 semantically incorrect links that resolve . The application introduces 600 request local citation references, accepts only registered IDs, and resolves them server side. Suppose 588/600 references successfully resolve to the correct authorized document/version, while 12/600 are rejected because the model emits unrecognized citation IDs or the user lacks current access. Verified correct citation delivery rate = 588/600 = 98% . Fabricated URLs rendered = 0/600 = 0% by construction: unresolved citations are blocked, not counted as verified . Do not report 100% citation coverage; it remains 98% because 12 cannot be displayed as verified links. If the benchmark includes 500 grounded claims , a separate span support judge finds only 460/500 = 92% supported by cited passages. Thus the 98% safe link resolution does not imply 98% claim faithfulness; this illustrates the distinction between link provenance and claim entailment . Example registry: ref A → (tenant=T1, doc=policy sg 2026, version=v3, section=4.6). A generator outputs citation ref ids=["ref A"], and the server returns a verified viewer route or permitted original URL. If it outputs ref Z or a fabricated https://.../2026, the renderer rejects it. Suppose p95 increases from 155 ms to 173 ms with registry/permission checks, an illustrative +18 ms. Cache safe reference metadata cautiously while performing current authorization checks. These counts and latency values are hypothetical , not MachinoAI production measurements. For a user who has lost access since indexing, ref A must fail closed even if the source still exists; availability is not authorization. 5. Engineering implementation Step A — preserve provenance at ingestion: assign immutable source/document IDs independent of file path or title; store version, content digest, connector native handle and verified external URL only when supplied by a trusted connector or repository. Step B — produce a retrieval manifest: after filtering and context packing, record precisely which authorized chunk spans were sent to the model. Include a request local allowlist of citation IDs and hashes; treat out of context citations as invalid. Step C — ask for structured references: request an answer with statement level citation ref ids drawn from the allowlist. Constrained schema output is helpful, but membership is validated separately on the server. Step D — validate IDs and support: reject unknown references, duplicate malformed IDs, references to dropped chunks, wrong version source IDs and unauthorized documents. Optionally verify the cited span supports the statement (Q268). Step E — resolve with a trusted registry: map IDs to canonical URLs or a secure document preview route. Use server generated URLs, not model output, and never accept hostnames/paths from the LLM as destinations. Step F — enforce safe link policy: only approved schemes (https for external destinations), expected domains/connector routing, no javascript:/data: or localhost/internal endpoints, and validated redirect behavior. Avoid URL fetches to arbitrary destinations to prevent SSRF. An allowlisted domain is a defense but not a substitute for source identity. Step G — handle absent URLs: return source title, section and a verified in product preview or a non clickable provenance label. Do not guess URL patterns from a document name. Step H — repair in bounded steps: if a generated citation ID fails, attempt targeted regeneration with the same frozen valid reference set or remove the unverifiable citation and mark the answer limited. Cap retries to avoid loops; never silently substitute a nearby document. Step I — secure cache and auditing: store answer id, manifest hash, citation ref id, source version, resolution outcome, authorization decision, original connector link source, final rendered URL and timestamp; exclude tokens/secrets and respect retention. Step J — deploy in phases: shadow resolve generated citations first, compare existing and resolved destinations, then canary deterministic rendering; monitor malformed IDs, rejection, broken link rate, wrong document rate, provenance mismatches and latency with a rollback that never reinstates unsafe free text URLs. 6. Failure scenarios and trade offs Valid looking but wrong link: URL syntax or HTTP 200 does not prove the linked document supports the claim. Verify document/version identity using registry metadata and optional source hash. Valid ID with no user facing URL: the source can be cited via controlled internal preview or title/section. “No URL available” should not be hallucinated away. Mutable file path: renames and folder moves break path derived URLs; use stable document IDs and a resolver (as in Q239). Wrong document behind redirect: a trusted source link may later point to another record; validate endpoint/identity and redirect policy, not only reachability. Short lived signed URLs: secure, but expire and should not be stored or surfaced to the model. Generate at click/display time from an authorized trusted service. Per request reference numbers: simple for generation, but merging results or reordering after assignment can swap citations. Freeze the manifest before calling the model. Constrained JSON: can prevent arbitrary URL fields and malformed output but cannot prove that an allowed reference actually supports the claim; add span level grounding validation. Model produced markdown: convenient but dangerous if arbitrary links auto render; use plain text plus separately bound validated citations. Link checks: eager network requests add cost and SSRF exposure; use registry metadata and controlled connector checks, not blind fetching of user/model supplied links. Citation strictness: blocking unresolved citations reduces false provenance but can lower visible citation coverage. Monitor rejections separately, and show honest “source unavailable” states. Identity vs location: even a canonical original URL may change after reorganization. Registry IDs, provenance and version contracts must remain authoritative. Privacy: URL paths or embedded identifiers may reveal confidential file titles. Render only what the user's current permission permits. 7. Prevention, testing and monitoring Add adversarial tests where the model invents plausible URLs, swaps reference IDs, uses [1] after rerank reordering, cites a chunk omitted from the final prompt, mixes versions, or requests a source belonging to another tenant. Inject document moves, renamed folders, index alias swaps, deleted documents, expired signed links and redirected destinations; verify stable ID provenance and current authorization. Build a golden set mapping answer claim → allowed citation ref id → source/document/version/span → expected URL/viewer behavior. Include intentional “no URL available” records. Assert zero rendered links from LLM provided URL strings , 100% resolved links originate from trusted registry entries , and explicit non success statuses for unsupported or unauthorized references. The fraction of reference IDs successfully resolved can still be below 100%. Measure fabricated URL exposure rate , verified citation resolution success, wrong document link rate, outdated version link rate, link unavailability, authorization violations, citation support accuracy and p95/p99. Compare model visible reference manifest with emitted IDs and server resolved sources under correlation IDs. Report which boundary first failed: source metadata, prompt manifest, generation, citation validation, resolver or UI. Track schema/connector drift and invalidation on source move, ACL update, source delete or restatement; do not rely on old cached document URLs. Human review sampled high impact claims and source previews to catch a link that is technically valid but evidentially irrelevant. Re run tests after prompt, model, retriever, connector or citation renderer changes. Interview takeaway: Never let an LLM generate source URLs. Give it a request local list of allowed citation IDs, validate every emitted ID, and deterministically resolve it through an authorized, version aware source registry. Render only trusted links; treat unsupported links as failures or unlinked provenance. Check citation identity and claim support separately so a plausible URL never becomes false evidence. Authoritative references Azure AI Search, Query Knowledge Base via API or MCP (references, doc keys and service generated citation URLs): https://learn.microsoft.com/en us/azure/search/agentic retrieval how to retrieve Azure AI Search, Create an index for agentic retrieval (citation metadata fields): https://learn.microsoft.com/en us/azure/search/agentic retrieval how to create index Azure AI Search, Agentic retrieval overview (grounding references and activity): https://learn.microsoft.com/en us/azure/search/agentic retrieval overview AWS Bedrock, Knowledge Bases data source references : https://docs.aws.amazon.com/bedrock/latest/userguide/kb test retrieve generate.html
A user asks a multi-part question, and the answer addresses only the first part. How would you determine whether retrieval coverage or generation is responsible?
Production incident (illustrative): An enterprise procurement assistant is asked: “What is the Singapore contractor hotel cap, who approves an exception, and what evidence must be attached to a reimbursement claim?” The answer states “SGD 350 per night” , cites the current contractor policy, and stops. It neither names the Regional Finance Director who authorizes a documented exception before booking nor explains the required written approval and supporting receipt. The cap is factually correct, and the citation is genuine, but the user requested three distinct outputs . The team cannot tell whether source retrieval failed to cover the approval/documentation clauses or whether the LLM ignored those clauses in its prompt. Core principle: Treat every meaningful subquestion as a traceable coverage obligation from user intent through retrieval, reranking, context assembly, generation, and displayed response. Ask where the first missing subquestion occurred . If evidence for part 2 was never retrieved, generation cannot legitimately answer it; if it was in the exact final model visible prompt, then the omission is a generation, answer budget or post processing issue. Microsoft Azure RAG guidance describes query decomposition for complex questions and separately evaluates response completeness—whether an answer resolves all aspects of a user's request. 1. Establish the question contract before changing retrieval Freeze the exact user question, conversation turns, region, entity, tenant/ACL, policy effective date, model, retrieval config and corpus snapshot. Clarify whether the three requested parts are independent or whether one depends on another. Decompose without answering into atomic intent slots. For this case: P1 = standard hotel cap, P2 = exception approver and timing, P3 = supporting documentation for reimbursement. Retain the original wording and referents for each part; the third part concerns reimbursement evidence , not generic onboarding paperwork. Make each subquestion self contained using authorized contextual identifiers: “For a Singapore contractor under the effective 2026 travel policy, what is the nightly accommodation cap?”; “Who can authorize a cap exception, and when?”; “What attachments are necessary for reimbursement?” Capture a reference answer checklist from current authoritative sources: P1: SGD350/night; P2: prior written Regional Finance Director approval when applicable; P3: receipt and written exception approval where relevant. These policy details and figures are illustrative ; validate actual enterprise policy before implementation. For each slot, store slot id, intent, expected fact set, source scope, source span ids, applicability, severity, dependencies, answerability. Preserve whether different policy sources or tools are required. Do not blindly split every query into many pieces. Single fact questions may be handled more efficiently by one search; multipart decomposition is justified when independent facts or evidence sources are likely. Establish that all requested parts are authorized to answer. If a part requires restricted employee data, absence of that unauthorized detail is not a retrieval completeness failure. 2. Build an evidence survival matrix per subquestion Source availability: verify an authorized, active source actually contains authoritative evidence for each slot, using SME labelled document/version/span identifiers. Planning coverage: check whether each slot received a query plan or was accidentally merged into a weaker “hotel cap” search. Log all planner outputs, source routes and reasons for omitting a subquery. Retrieved candidate coverage: measure which slots are supported anywhere in raw dense/lexical results before fusion. An answerable part missing here is a search/planner/source problem. Reranked coverage: inspect candidate selection and fusion. The right document may have been retrieved but an exception clause could be pruned by top k, source grouping or duplicate suppression. Final context coverage: inspect the exact serialized prompt sent to the reader, including citation IDs and source offsets. Verify all relevant snippets survived compression, token budgeting, metadata serialization and final prompt ordering. Generator obligation coverage: extract answers at the granularity of P1, P2 and P3 from the model's full raw output, not merely visible summary text. Measure correct, grounded completion or explicit justified abstention for each. Displayed response coverage: inspect response post processing, length caps, streaming termination, UI summarizers and citation rendering; text may be generated correctly and then hidden or truncated. Assign a first loss reason code per slot : SOURCE ABSENT, PLAN OMITTED, RETRIEVAL MISS, RERANK DROPPED, CONTEXT TRIMMED, GENERATION OMITTED, OUTPUT TRUNCATED, UNANSWERABLE or UNAUTHORIZED. One request may have a different cause for each slot. 3. Controlled experiments to separate retrieval and generation Experiment A — query decomposition audit: hold the corpus/index fixed. Compare an original composite query against individually searched P1, P2 and P3, using the same filters, source authorization and candidate budgets. If P2/P3 succeed individually but not in the composite plan, the problem is planner/query dilution, not missing indexed content. Experiment B — oracle evidence injection: bypass real retrieval for one controlled test and supply exact authorized gold spans for all three slots to the original answer model, preserving source metadata. If the model still answers only P1, generation/prompt instructions/response length are implicated. Experiment C — frozen final prompt replay: hold identical serialized system/user messages and the same P1/P2/P3 evidence, generation settings and model revision; repeat generation. If P2/P3 are inconsistently omitted, inspect decoder variation and prompt overload. If always omitted, inspect instructions, task decomposition and output schema. Experiment D — fixed generator with retrieved prompts: replay actual candidate lists and their assembled contexts through the same model with the same settings. Compare whether missing slots track missing evidence or persist when the evidence is present. Experiment E — expanded retrieval only: improve coverage for P2 and P3 using targeted subqueries but do not change the generation prompt. If final completeness improves when new evidence is included, retrieval was a limiting factor—though generation can remain independently imperfect. Experiment F — generation only: keep the same complete, authorized evidence set and add explicit answer slots/coverage validation. If completeness improves without retrieving anything new, generation or answer validation was a limiting factor. Experiment G — interactions: test both targeted retrieval and structured generation in a paired 2×2 evaluation; source availability, position effects and token limits may interact. An observed gain from retrieval does not prove the decoder was perfect. Safety: never manufacture oracle evidence in production. Oracle injections are labelled offline tests using permitted, verified source spans; no evaluation result should be interpreted as a production source. 4. Root cause decision tree with production examples P2 never planned: intent parser interpreted the complete question as a single hotel cap lookup. Fix decomposition, check that every explicit conjunction (“and,” “also,” “plus”) yields appropriate slots, and run distinct subqueries. P2 planned, no matching source candidates: search rewrite, source routing, wrong tenant/filter, embedding dilution or stale index. Validate per slot exact/lexical retrieval and authorized source completeness. P2 candidate found, discarded in fusion/rerank: diagnose a common policy overview dominating the top N and a specific approval clause ranking lower. Apply per slot evidence coverage constraints or source/facet aware selection. P2 passage retrieved and selected but absent from serialized prompt: context trimming, dedup or compression. Repair span survival as in Q266, with source offsets and token budget tests. P2 present in final prompt, answer still omits it: model neglects the second intent, adheres too strongly to “answer in one sentence,” suffers from context position/noise, or makes an incomplete synthesis. Test slot based generation and Q267 style position controls. P2 correctly expressed, P3 missing: investigate P3 independently. Do not assume one root cause for all missing parts. All parts present in raw generation, only P1 displayed: detect API output length, UI formatting/collapsing or streaming stop. Fix rendering/transport, not retrieval. Evidence missing for a genuinely unanswerable part: say so explicitly; a response can be complete about the user's request by acknowledging a missing authoritative answer, but this must not be counted as fully factual coverage. Wrong facts joined across parts: bind entity, role, effective date and source version; a part 2 answer from employee policy cannot be grafted onto a contractor question. One question asks for an overall comparison: answering each component independently may still fail to synthesize the comparison. Preserve cross slot dependencies and add a final comparison/decision obligation. 5. Numerical example: stage by stage missing part attribution A labelled test set contains 100 three part questions = 300 answerable subquestion slots , all with verified authorized evidence in the frozen corpus. Assume no abstention/ACL exceptions for this experiment. Baseline evidence funnel: planning includes 280/300 slots; retrieval finds evidence for 255/300; reranking preserves 240/300; the exact final prompt contains sufficient evidence for 225/300; final answers correctly and supportably address 180/300. These are nested counts, and first loss totals are planning 20 , retrieval 25 , reranking 15 , context assembly 15 , and generation 45 — 120 missing slots altogether. Thus retrieval related early loss (planning + retrieval + selection + context assembly) accounts for 75 missing slots ; generation side omission accounts for 45 . The difference matters because adding an instruction to “answer all three parts” cannot restore 75 absent evidence slots. Illustrative production incident: composite retrieval finds P1 only; running each authorized subquery individually finds P1, P2 and P3. With the original prompt and oracle spans for all three, the model answers P1 and P2 but omits P3— both query dilution and generation incompleteness are present. Apply query decomposition, per slot candidate coverage and protected evidence packing. In a controlled rerun on the same 300 slots, suppose the prompt contains evidence for 285/300 = 95% instead of 225/300 = 75% . With a structured answer obligation ledger, correct grounded slot completions rise from 180/300 = 60% to 270/300 = 90% , an absolute +30 percentage points . The remaining 15 absent evidence slots cannot be answered from the provided prompt; another 15 have evidence but are still missed or misanswered by generation. The all three parts complete rate is measured separately: suppose 32/100 = 32% at baseline and 80/100 = 80% after repairs. You cannot derive this question level figure solely from average per slot recall, because missing facts may cluster within the same questions. A planned 3 query fanout increases per request retrieval calls from 1.0 to 2.4 on average, and p95 end to end latency from 210 ms to 340 ms , hypothetical +130 ms . Decomposition and extra validation should be conditional on multipart intent and constrained by budget. All figures are illustrative only , not MachinoAI measurements or empirical vendor benchmarks. Validate exact labels, confidence intervals, source freshness, latency and monetary cost in actual canary cohorts. 6. Engineering solution: slot aware retrieval and synthesis Intent decomposition contract: parse each multipart query into a stable list of required intents and dependency edges. Preserve original entity, user scope, negation, fiscal date, units and comparison instructions during decomposition. Coverage oriented planning: assign each slot an authorized source set, query variant, expected type of evidence and budget. When source routing fails, do not silently use a topically similar but non authoritative source. Parallel targeted searches: run independent subqueries when useful, but share snapshot, tenant, ACL and effective date constraints. Azure's agentic retrieval describes decomposition into targeted parallel queries and returning grounding references/activity for traceability. Per slot evidence inventory: attach slot id → [source id,version,section,span] to each supported candidate. One passage may support several slots; keep it once and map all applicable slots to avoid waste. Protected packing: reserve token budget for at least one current authoritative span per answerable slot before filling the remaining budget with ancillary context. If budget is inadequate, record missing slots and use bounded refetch, hierarchical synthesis or transparent partial answer. Structured synthesis: produce an internal object {P1:{answer,evidence ids,status}, P2:{...}, P3:{...}}; the display layer can render concise paragraphs/bullets. Do not let the model omit a key entirely without returning unsupported/unknown and a reason. Deterministic coverage gate: compare output keys against the original slot ledger; ensure each answer is actually supported by its cited spans and that users are told about unanswered components. A valid slot ID is not enough if its text is wrong. Grounding verifier: run exact checks for numbers, role, dates and source IDs; use an evaluated entailment checker for paraphrased claims. Link the answer to authorized model visible evidence. An unsupported answer should be blocked rather than filled from model memory. Bounded repair: for one missed slot, retry only the targeted authorized retrieval or regenerate from a fixed verified context; do not endlessly rewrite the whole answer or broaden to forbidden sources. Observability: log query id, slot id, planner status, retrieval found, rerank kept, prompt included, generation covered, source version, citation support, failure reason to attribute loss. Avoid storing sensitive raw user text by default. Safe canary: compare slot level and all slots coverage against a gold set; gate on groundedness, wrong source citations, false “complete” assertions, p95/p99, token costs and ACL protection. Feature flag decomposition and the answer contract separately to isolate their effects. 7. Trade offs and common mistakes Decompose every query: raises cost and can damage simple questions. Use intent detection and preserve dependency structure. Split into too many subquestions: causes excessive calls, duplicate evidence and extra joins. Deduplicate semantically identical slots, but never merge distinct requested outcomes. High top k without decomposition: may return many near identical P1 paragraphs while ignoring P2/P3. Evaluate per part evidence coverage, not only average query relevance. Generation checklist alone: can make empty promises for unseen evidence. Mark missing slots as unavailable rather than hallucinating. Oracle injection misinterpretation: if gold evidence fixes the answer, generation may still fail with noisier real contexts; benchmark both oracle and realistic retrieval conditions. Groundedness vs completeness: high faithfulness for the one stated cap cannot substitute for addressing the approval and receipt parts. Measure both dimensions. Longer answers: can include more facts but may increase unsupported claims, verbosity and latency. Optimize correctly supported obligations per response. Strict all slot success: important for procedural tasks but can unfairly penalize safe explicit abstention when a source truly lacks an answer. Report complete factual answers and honest part level uncertainty separately. Multi source authority: evidence from an unrelated HR policy cannot fill a vendor contract slot because of text similarity. Enforce source relevance and authorization. Asynchronous fanout: parallel queries can return on different index revisions; pin source snapshots and document applicability to avoid stitched inconsistent facts. Generation model stochasticity: a single oracle run does not establish deterministic coverage. Repeat frozen prompt trials and assess material omissions, not text identity. UI level truncation: if output exists but isn't displayed, expanding retrieval or changing prompts cannot solve it. Statistics: a 90% slot success average can hide a much lower all parts complete rate. Report both, confidence intervals and high severity missing obligations. 8. Production monitoring and regression tests Maintain expert labelled multipart questions with two, three, and five components; include mixed source types, prerequisites, comparisons, numeric versus procedural answers, negative constraints, nested clauses and genuine unanswerable parts. Add subquestion planning recall , per slot retrieved evidence Recall@k , rerank and final prompt survival , per slot supported answer rate , all required slots complete rate , unanswered slot disclosure, wrong source citations and cost. Inject faults separately: omit P2 from planner; remove P3 from candidate list; clip P2 during packing; hide P3 near middle; force low output tokens; drop the generated P2 block in post processing. The first loss telemetry should identify each injected defect. Run offline fixed corpus retrieval only , oracle context generation only and end to end evaluations on every material query planner, source routing, reranker, model, prompt, tokenizer or context budget change. Review a sample of automated slot extraction and coverage labels with domain SMEs. LLM judges can miss negation, role, time and modality; evaluate false positives when detecting apparent “completion.” Guard security: disallow retrieving an unauthorized source merely to make all slots appear complete, and invalidate cached slot evidence when documents are retired or ACLs change. Canary with rollback conditions based on critical missing part escape rate, wrong or fabricated answers, unexpected source fanout and p95/p99 regression. Keep user facing response concise: show each requested part and, where evidence is missing, explicitly state which part could not be verified. Interview takeaway: If a multipart RAG answer addresses only the first part, decompose the question into answer obligations and trace each one through planning, retrieval, reranking, prompt assembly, generation and rendering . Compare subquery retrieval with oracle context and frozen prompt tests to prove whether a specific part lacked evidence or the model failed to use it. Fix the first failing boundary; ensure every part receives a supported answer or an honest, explicit uncertainty status . Research references Microsoft Azure Architecture Center, RAG information retrieval and decomposition: https://learn.microsoft.com/en us/azure/architecture/ai ml/guide/rag/rag information retrieval Microsoft Azure AI Search, Agentic Retrieval Overview: https://learn.microsoft.com/en us/azure/search/agentic retrieval overview Microsoft Azure Architecture Center, RAG End to End Evaluation and Completeness: https://learn.microsoft.com/en us/azure/architecture/ai ml/guide/rag/rag llm evaluation phase Microsoft Foundry, RAG Evaluation—Groundedness vs Response Completeness: https://learn.microsoft.com/en in/azure/ai foundry/concepts/evaluation evaluators/rag evaluators Amazon Bedrock, Knowledge Base Evaluation Metrics: https://docs.aws.amazon.com/bedrock/latest/userguide/knowledge base evaluation metrics.html
Your RAG API performs well for 10 concurrent users but becomes unstable at 1,000. How would you load-test and identify the first bottleneck?
Production incident (illustrative): A customer support RAG endpoint works for 10 employees: p95 end to end latency is 1.8 seconds, few errors, and answers cite the right product manual. After a launch exposes the same endpoint to 1,000 connected users, p95 exceeds 9 seconds, 429/503 responses appear, retries multiply traffic, and a few answers become incomplete after timeouts. Engineers initially blame the LLM because generation is the most visible stage, but distributed traces reveal an exhausted vector database connection pool and growing retrieval wait queue. The problem is neither fixed by increasing the API server pod count nor accurately characterized by saying “1,000 users equals 1,000 RPS.” Core principle: Model a RAG request as a queueing network—gateway → authorization → query preprocessing/embedding → vector/lexical search → rerank/context assembly → generator/provider → streaming response, plus caches, storage, pool limits and asynchronous dependencies. Recreate the arrival process, request mix, prompt/output token distributions, think times and concurrency ; observe each queue and stage before scaling. Distinguish users , in flight requests , arrivals per second , completed requests per second , tokens per second , and sustainable capacity . A service with 1,000 users may see very different workloads depending on how frequently they ask questions and how many tokens each answer consumes. 1. Define service objectives, risk limits and realistic traffic Capture the current accepted baseline at 10 concurrent users and select explicit test SLOs : for illustration, p95 full response latency ≤3 seconds, p99 ≤6 seconds, error fraction below 1%, zero unauthorized results and acceptable grounded answer accuracy. These are example targets, not universal standards for RAG workloads. Separately define time to first token (TTFT) , time to complete streamed response, time to first retrieved context, and a non streaming end to end duration. A fast TTFT with a 12 second tail is not a 3 second full answer. Recreate realistic user behavior: login/auth path, tenant and policy filters, bursty arrivals, session histories, question complexity, cache hit versus cold queries, top k, number of candidate sources, prompt length, answer token count and streaming/cancellation behavior. Distinguish 1,000 connected users from 1,000 simultaneously outstanding requests , and from 1,000 requests/second . Estimate request rate using measured think times and completion times, then test both realistic sessions and defined arrival rate limits. Build a sanitized representative workload: short factual lookups, multi part questions, long documents, cold/warm caches, complex reranking, large outputs and negative queries. Include authorized tenant diversity and avoid actual customer data in shared staging. Establish cost and safety budgets before stress tests. Provider calls consume paid tokens, external QPS quotas, shared inference capacity and potentially trigger vendor protections; never flood a live shared tenant without a controlled window and stop conditions. Record revisions of API image, prompt, embedding/rerank/generator models, vector index, source snapshot, connection pool settings, replicas and rate limits. Only compare runs with controlled changes. 2. Choose the correct load generator and test shape Closed loop user model: a virtual user sends a request, waits for completion, thinks for a specified interval, then sends another. It models many human sessions but its arrival rate falls as the system slows , potentially concealing overload (coordinated omission). Grafana k6 distinguishes these closed models from open arrival rate models. Open loop arrival model: schedule requests independently of the preceding response duration, e.g., k6 constant arrival rate and ramping arrival rate. Use it to find capacity under a stated target RPS. Preallocate sufficient test VUs and monitor dropped iterations so the generator is not mistaken for the service bottleneck. Concurrency stress: separately hold 10, 50, 100, 250, 500 and 1,000 in flight requests using bounded requests and controlled durations. This tests socket, memory, admission and per request resource limits, not necessarily realistic arrival rate. Step load: increase actual offered load in plateaus (illustratively 5→15→25→35→45 requests/sec), holding long enough to reach an approximate steady state or observe persistent queue growth. Restore baseline between comparative experiments when appropriate. Spike/soak: test short bursts, gradual long running load, and recovery after load stops. Soak testing catches memory leaks, stale connections, growing caches and accumulating retry/dead letter queues that a 2 minute burst misses. Cold versus warm: run an index/query cache cold test and a warm repeat test; preserve both in reports. A 100% cache hit benchmark should not be extrapolated to production's unique user questions. Loader checks: distribute the load generator across hosts or workers when needed (Locust supports master/worker distributed execution), and report its CPU, memory, open sockets, network egress, VU saturation and scheduled versus actually sent requests. Ground truth safety: use a fixed authorized golden question subset and compare citation validity, answer completeness and decision accuracy under load; HTTP 200 without a usable answer is not success. 3. Instrument one end to end trace and each critical queue Propagate one request trace ID across gateway, query rewrite, embedding API, vector and lexical retrieval, source filters, reranking, packing, generation, token streaming and response formatting. Keep tenant id pseudonymized and avoid logging document bodies by default. For each stage record arrival, queue enter, work start, work end and completion timestamps if possible; separate wait time from service time. A 4 second vector “duration” may contain 3.5 seconds of pool waiting and 0.5 seconds of actual ANN search. Capture stage p50, p95 and p99; throughput; in flight tasks; concurrency permits; worker/thread event loop utilization; saturation; queue length and age; timeout/cancellation; retries; and error codes. Do not add stage p95 values to claim an exact end to end p95. At the gateway watch active connections, CPU, event loop lag, request backlog, memory, load balancer queue, connection resets and tail latency. Detect whether a client timeout is occurring while the backend continues working. At embedding/provider layers watch in flight calls, total/input/output tokens, quota counters, 429, retry after, retries per user request, latency and billable tokens. Batch/caching strategy changes cost and throughput; measure both. At vector search watch connection pool occupancy and wait, ANN execution latency, shard/replica CPU, I/O, disk/page cache, index memory, filter selectivity, result completeness, network hops and shard queue depth. At reranking and context assembly watch candidate counts, batch sizes, GPU/CPU saturation, tokenization time, memory and context window packing. Heavy reranking can overtake vector search even when raw search stays fast. At LLM generation watch provider request latency, TTFT, output tokens/sec, input tokens, output tokens, concurrency slots, quota errors, and whether a long streaming request continues to occupy an admission permit. Use OpenTelemetry style tracing and versioned semantic conventions where useful; correlate per stage timing with token usage and resource saturation. Some GenAI metric names evolve; validate against deployed instrumentation library/version rather than assuming a static schema. Record offered vs accepted vs completed RPS , errors by component, dropped tests, and outstanding backlog. An increase in 200 responses per minute does not prove healthy throughput if offered demand and error queues are growing. 4. Root cause debugging: find the first bottleneck A. Establish first unstable plateau: graph offered RPS, completed RPS, p95/p99, 429/503 and queue depth at each step. The knee occurs where throughput stops rising roughly with offered load while queues and latency start growing. B. Inspect earliest growing queue: compare gateway, embedding semaphore, vector connection pool, reranker slots and model provider admission. The first stage with sustained backlog and high utilization is often the immediate constraint; a later downstream queue may be a symptom. C. Examine work start timestamps: a vector API span that looks slow may mostly wait for a free DB connection. Separate queue wait from vector execution before tuning HNSW, k or index parameters. D. Check upstream amplification: retries after 429/503, speculative parallel retrieval, duplicate queries, user cancellation not propagated, and client reconnect can multiply requested work. Measure attempt rate per original request and bound amplification. E. Use component isolation: replay queries with a stub embedding result, fixed vector candidates, bypassed reranker, or deterministic mock generator only in an isolated test . Observe which intervention moves the throughput knee. A mock LLM might hide its token rate bottleneck; interpret each test narrowly. F. Inspect server side pressure: API autoscaling based only on CPU can miss exhausted DB pools or external quota limits. Scaling from 4 to 20 API pods can multiply connection demand and worsen vector database contention. G. Verify external saturation: if embedding/LLM providers show high queue/429 rates, adding local workers will not raise sustainable provider capacity. Measure requests/minute and tokens/minute against the actual assigned limits. H. Separate slow dependencies: an extra retrieval source may introduce network fan out or head of line blocking; Q278 will cover multi source critical path optimization specifically. Q279 will cover slow external API deadlines; Q277 will cover spike driven embedding 429 and retry budgets. I. Validate quality under load: partial retrieval due to deadlines or throttles can cause wrong citations, stale cache fallback or omitted exception clauses. Correlate answer quality failures with load stage rather than treating performance and correctness as independent. J. Identify instability type: queue saturation, throughput collapse, timeout, crash/OOM, container restart, provider quota, unhealthy replica or client generator overload each needs different remediation. Do not infer one bottleneck from aggregate CPU. 5. Beginner friendly capacity and queueing numerical example User math: imagine 1,000 active users who, on average, spend 20 seconds thinking and each request takes 4 seconds end to end. A simple closed interaction estimate yields 1000/(20+4) ≈ 41.7 requests/sec; 1,000 users is not 1,000 RPS . Real arrivals are bursty, so test more than this mean. Stage capacity (illustrative): embedding 100 RPS; vector search has 16 effective concurrent connection slots , with mean active service time 0.5 sec , giving an idealized upper bound 16/0.5 = 32 searches/sec; reranking 60 RPS; generation provider supports 50 RPS at this specific input/output token mix. Thus vector connection admission is the lowest approximate ceiling: 32 RPS . Step observations (hypothetical): 5 RPS → p95 1.8s, 0% errors; 15 RPS → p95 2.0s, 0%; 25 RPS → p95 2.6s, 0%; 35 RPS → p95 8.2s, 2% errors; 45 RPS → p95 14.5s, 8% errors. At 35 RPS, completion throughput hovers around 31–32 RPS while connection wait and outstanding queue keep growing. Queue buildup: if 35 requests/sec arrive and vector admission can sustain 32/sec , the idealized undrained backlog increases by 35 32=3 requests/sec or 180 waiting requests after 60 sec before timeouts, shedding and service time variation. This approximation explains why latency can deteriorate before CPU looks fully used. Observed trace: at 35 RPS, vector span p95 is 4.4 sec , including 3.8 sec pool wait and only 0.6 sec search ; embedding remains p95 90 ms, reranking p95 140 ms, generator p95 1.1 sec. High vector waiting , not raw vector compute, is the immediate first saturation. Fix test: raise safe vector capacity by optimizing pool/admission, query execution, shard capacity or adding appropriately provisioned replicas; benchmark a revised sustainable vector service ceiling of 48 RPS under the same mix and with measured accuracy. Then re run all plateaus: if stable p95 at 35 RPS becomes 2.7s with <1% errors, the bottleneck has moved or been reduced. Do not claim that merely opening more connections always raises capacity; extra connections can overwhelm the database. New ceiling: generation at 50 RPS may become the next limiting service, and total demand around 45 RPS could still approach its token/minute or concurrency ceiling when prompts get longer. Capacity planning must consider token throughput as well as request throughput. Little's law check: at a stable throughput of 25 RPS and mean time in system 2 seconds, average in flight work is roughly L = λW = 50. This average relation assumes a stable system; it is not valid to extrapolate indefinitely from a queue that is growing without bound. Quality check: suppose grounded correct answers are 97/100 at 10 users but 91/100 at peak due to failed source calls and incomplete fallback contexts. Performance release criteria must include grounded answer quality , not just p95 or success HTTP codes. All figures are illustrative rather than measured MachinoAI telemetry. The service capacity calculation assumes stable average service time; measured queueing behavior, tail latencies, shard contention and provider quotas can lower real capacity. 6. Engineering solutions in recommended order First improve measurement and isolation: add queue time spans and dashboards; validate load generator capacity, realistic token mix and thresholded tests before changing infrastructure. Admission and concurrency control: place bounded per stage semaphores/queues in front of scarce vector connections, reranker GPUs and LLM calls. Reject or defer excess work predictably instead of permitting unbounded memory, queue age and timeouts. Capacity scaling: right size vector pool relative to shard/server limits, scale retrieval replicas where justified, optimize slow filters and index placement, tune rerank batch/concurrency and allocate worker CPU/memory after identifying the actual constraint. Provider aware quotas: cap upstream embedding and generation request/tokens per minute, distinguish 429 from 5xx, use bounded retry after aware backoff and per original request budgets. Q277 explores embedding spike backpressure and retries in detail. Cancellation propagation: abort or stop expensive downstream work when user request has expired/cancelled, where the provider and client stack support it. Avoid doing retrieval or generation that cannot be returned. Cache responsibly: cache authorized embedding results for repeated normalized queries, shared public facts where permitted, and version scoped document metadata. Include tenant, ACL, source version and effective date in appropriate cache keys, and never use cross tenant caches that leak results. Fairness: per tenant quotas and weighted scheduling prevent one large tenant or expensive long context query from starving other users. Separate interactive and batch queues. Cost aware token controls: limit irrelevant context and enforce measured output budgets, but preserve critical evidence; token trimming that drops policy exceptions may reduce latency while making answers incorrect. Graceful overload: choose bounded queuing and clear 429/503 responses where necessary; do not return fake “success” responses with unsupported or incomplete claims. Scale in stages: after each controlled improvement, rerun the same load mix, then identify the next bottleneck. Scaling one stage moves the knee rather than eliminating physical limits. 7. Trade offs, test pitfalls and failure modes Closed loop only: underestimates high offered traffic when latency rises because users wait; add open loop tests to avoid coordinated omission. Open loop only: may not model users' natural think times and cancellations; compare both modes with explicit definitions. One thousand VUs: does not imply 1,000 simultaneously active API requests or 1,000 RPS. Publish actual scheduled/started/completed rates. Short stress only tests: miss queue buildup, connection leaks and sustained provider quota depletion; include steady state and soak durations. Too few load generator VUs: k6 dropped iterations can result from inadequate VU preallocation; report generator utilization and errors separately from server overload. Aggressive retries: can turn a manageable transient fault into positive feedback and worsen the overloaded stage. Limit attempts and consider load shedding. More application pods: increases upstream concurrency and connection pressure; provision constrained downstream dependencies in tandem. Larger pools: can reduce waiting only if the backend has spare capacity; otherwise they can raise contention and tail latency. Caching: improves hot query throughput but can hide cold query bottlenecks and leak restricted data if cache authorization is wrong. Batching: improves throughput on embeddings/reranking but can worsen latency for a single urgent request; measure both. Provider tokens/minute: a 50 RPS quota does not guarantee 50 RPS effective throughput for long prompts; track tokens/minute and streaming concurrency. Latency percentiles: summing stage p95 values is not mathematically equal to end to end p95. Use per request critical path traces. SLO versus correctness: response time wins achieved by incomplete source fetching, truncated citations or silent fallback may fail the user. Keep retrieval coverage/groundedness gates. Load testing in production: can exhaust third party quotas or disrupt unrelated users. Use isolation, budget caps, rate safety rails, phased windows and explicit shutdown triggers. 8. Prevention and operational release gates Maintain a versioned load test workload covering short and long prompts, retrieval only and fully generated responses, cold/warm caches, streaming, tenant filters, concurrent sessions and high token queries. Run ramp/step/spike/soak tests at every material RAG pipeline release; require p50/p95/p99, time to first token, full response latency, accepted/completed RPS, offered load, error/429/503 rates, queue age, dropped iterations and cost. Define a maximum sustainable throughput as the highest tested arrival rate where queues remain bounded, output quality meets minimums, the error/latency SLO passes and load can be held without accumulating debt. One fast burst does not establish a stable capacity limit. Add synthetic bottleneck scenarios: embedding provider 429, vector pool exhaustion, slow rerank batch, low model token quota, dropped shard replica, downstream cancellation and load generator CPU saturation. The stage local telemetry should identify each. Track pool occupancy/waits, concurrency permits, memory, event loop lag, provider request/token quotas, per tenant fairness and missing authoritative citations at overload. Periodically recalculate capacity when embedding/generation models, token budgets, document sizes, index configuration, user mix or provider limits change; a new model snapshot can move the bottleneck. Use canary and feature flags for concurrency, cache, replica and batching changes; retain a safety rollback path that respects tenant ACLs and current source validity. Set an operational alert for rising queue wait while service time remains flat —an early sign of a saturated admission pool—and a separate alert for service time inflation indicating CPU, index or provider degradation. Keep a short incident runbook: identify first affected request cohort → compare offered/completed load → find earliest growing queue → verify actual backend service capacity → isolate/replay → fix one constraint → rerun exact workload → confirm quality, cost and SLOs. Interview takeaway: A RAG API that works for 10 concurrent users but fails at 1,000 needs a controlled open/closed loop load test with per stage queues and traces , not immediate autoscaling. Convert real user behavior to offered RPS, ramp load to the first throughput knee, locate the earliest queue whose wait grows , then test one bottleneck fix at a time. Verify that improvements preserve grounded answers and do not merely hide failures behind caches or dropped requests. Research references Grafana k6, Open and Closed Models : https://grafana.com/docs/k6/latest/using k6/scenarios/concepts/open vs closed/ Grafana k6, Constant Arrival Rate : https://grafana.com/docs/k6/latest/using k6/scenarios/executors/constant arrival rate/ Grafana k6, Dropped Iterations : https://grafana.com/docs/k6/latest/using k6/scenarios/concepts/dropped iterations/ Grafana k6, API Load Testing and Thresholds : https://grafana.com/docs/k6/latest/testing guides/api load testing/ Locust, Distributed Load Generation : https://docs.locust.io/en/stable/running distributed.html OpenTelemetry, GenAI Semantic Conventions and Metrics : https://github.com/open telemetry/semantic conventions genai/blob/main/model/gen ai/metrics.yaml
A traffic spike causes embedding API rate limits and cascading retries. How would you introduce backpressure and retry budgets?
Production incident (illustrative): At 09:00, a customer support RAG API receives a burst of 180 new questions per second after an internal launch. Each question triggers one embedding call before vector search. The organization’s embedding allocation is 6,000 requests/minute (100 requests/second averaged) and 1.2 million input tokens/minute ; these values are fictional test settings, not provider limits. A typical question consumes 300 embedding tokens. Demand is 10,800 calls/minute and 3.24 million tokens/minute, exceeding both budgets. The embedding provider starts returning HTTP 429. The SDK silently retries twice; the RAG service also retries twice; the frontend retries failed requests. Work multiplies, queues grow, and p95 latency goes from 1.6 to 14 seconds. Users repeatedly resubmit while the system is still processing canceled attempts. Root cause: Multiple independent retry loops turned a rate limit response into retry amplification . The system lacked shared request/token admission budgets, a bounded waiting queue, deadline awareness and an explicit owner for retries. Increasing worker count made the provider overload worse. Fix this before tuning embedding similarity, vector indexing or model generation. What this lesson covers: Q219 measured embedding latency, Q233 covered backpressure for offline ingestion , and Q276 established load testing and bottleneck identification. Q277 focuses specifically on online query embedding provider quotas, admission control, Retry After, retry budgets and overload containment ; Q278–280 will cover retrieval fan out, slow dependencies and latency budget allocation. 1. Diagnose whether this is a retry storm or a genuine quota change Gather one user request correlation ID and all upstream/downstream attempt IDs : gateway → RAG orchestrator → embedding client/SDK → provider. Count total provider attempts per unique original user request, not just HTTP 429 responses. Inspect provider error status and structured error code . Some 429s represent temporary rate limiting; others can indicate project quota or spend limit failures requiring operator action. Do not blindly retry every 429. Temporary 503 overload may be retriable, but invalid credentials, insufficient quota, unsupported models and permission errors are not transient backpressure signals. Record requested/remaining RPM and TPM, per model or shared limits, Retry After, rate limit reset values where supplied, request timestamps, token estimates, model, project, account/tenant, effective quota and SDK retry configuration. Limits can apply at multiple organizational levels or to shared model pools. Measure offered originals/sec , accepted originals/sec , embedding attempts/sec , successful embeddings/sec , 429/503 rate, concurrent calls, queue depth and oldest queued age, per attempt and end to end latency, deadlines exceeded, client cancels and spend. Correlate the start of latency growth with (i) provider limit exhaustion, (ii) repeated requests from clients, (iii) application retry middleware, (iv) SDK retries, and (v) a backlog persisting after incoming traffic falls. Pin a test traffic mix: normal and very long inputs, short/long prompt token estimates, cache hit distributions, tenant identities and request bursts. RPM may be below the limit while TPM is exceeded, and vice versa. Capture one stable provider/model revision during replay and run a small controlled rate ramp with a test budget. Never recreate a retry storm against a shared production account. 2. Draw a single retry ownership boundary Pick one layer to own the embedding retry decision. Usually the RAG embedding adapter has the best knowledge of the total request deadline, quota, token cost and cancellation state. Disable SDK automatic retries when application retries are centrally managed, or explicitly count SDK attempts inside the same per original budget. A wrapper with max retries=2 surrounding an SDK with max retries=2 may cause up to 3×3 = 9 embedding attempts for one logical call, before frontend retries. Client/browser retries must use the same request identity and be constrained by server issued retry policy, especially if the previous request may still be running. Idempotency/deduplication is useful for request accounting but cannot make an embedding operation free. Classify errors as TRANSIENT RATE LIMIT, TEMPORARY OVERLOAD, PERMANENT QUOTA, AUTH OR PERMISSION, BAD REQUEST, CANCELLED, or DEADLINE EXPIRED. Never schedule retries for known permanent cases. Track a total retry budget : maximum attempts per logical request, absolute deadline, total allowed retry load relative to new traffic, and cost/token limits. An allowed attempt count is not permission to violate other budgets. If a provider reports a valid Retry After of 30 seconds but the original user's request expires in 4 seconds, do not retry after 4 seconds . Return a clear rate limit/unavailable response or route noninteractive work to a permitted deferred queue. Honor Retry After as a minimum delay, including its valid HTTP date or delta seconds form when applicable; add jitter and avoid synchronized wakeups. If absent, use capped exponential backoff with jitter. Pass one absolute request deadline through every dependency and cancel delayed tasks when no longer useful. A per attempt timeout is not the same as an end to end deadline. Use a single idempotent logical work identifier across orchestration stages so metrics and budgets count originals versus attempts unambiguously. 3. Enforce admission control before embedding work Dual rate budget: meter requests and input tokens independently. Admit an embedding only when both counters/credits permit it. Respect provider specific quota semantics instead of assuming universal rolling windows or token accounting. Concurrency gate: cap in flight provider calls using a semaphore or adaptive limiter, independent of RPM/TPM. Even a request technically within the average rate budget can cause concurrency pressure during a burst. Token estimate: compute or conservatively estimate input tokens, reserve the expected cost at admission and reconcile actual usage when reliable provider accounting is available. Overestimating reduces utilization; underestimating produces avoidable TPM violations. Burst allowance: use a token bucket or sliding window limiter with bounded burst capacity. A token bucket can approximate a configured budget, but provider windows and shared quota enforcement may differ; calibrate against observed headers and 429s. Bounded queue: enqueue only work likely to finish before its deadline. Enforce maximum queue length and age. Do not create an unbounded async task for every rejected incoming user query. Load shedding: when no safe queue slot remains, quickly return a standard overloaded response (such as 429/503 as appropriate) with a bounded client retry hint, rather than consuming more memory and upstream attempts. Prefer an honest unavailable response to a fabricated answer without retrieval evidence. Fairness: reserve capacity for interactive user queries and prevent background bulk embeddings from exhausting the online quota. If bulk ingestion and online queries share credentials or a provider quota, impose explicit class budgets, priority and per tenant caps. Cache with authorization: deterministic query embedding caches can avoid repeat provider calls when text normalization, embedding model/version, configuration and data handling requirements match. Separate embedding reuse from document retrieval authorization ; cached embeddings must never bypass tenant ACLs. Batch carefully: where a provider supports batching, combine eligible offline or short requests to reduce request overhead, but track total token consumption; batching can lower RPM while leaving TPM unchanged and increasing interactive waiting time. Adaptive rate control: use provider limit headers, 429 trend, Retry After and observed successes to decrease admission quickly under overload and increase gradually after recovery. Apply rate updates at the appropriate shared quota scope, not independently per API pod. 4. Retry scheduling algorithm and operational safeguards On each original request, compute deadline at, attempts remaining, tokens reserved, quota scope, and retry budget available. Before each attempt, check cancellation/deadline, then acquire both request and token credits plus a concurrency permit. Never hold scarce provider permits during a backoff sleep. On success, update rate limit estimates, metrics and cache; release permits in a finally block. Record actual provider attempt count. On temporary 429/503, inspect Retry After and structured error reason; backoff with random jitter, but retry only if the projected start+service duration fits the original deadline and global retry budget. On permanent quota/billing/auth errors, fail quickly and alert the appropriate owner. Retry loops cannot create budget or access. Under sustained high 429 rates, invoke client side throttling/circuit like admission reduction and stop retrying when overload is widespread . Avoid a circuit breaker that masks unrelated healthy scopes; key the controller to actual provider/model/quota scope. On timeout or cancellation, don't unconditionally resubmit a request whose provider side completion status is unknown. Account for possible orphaned work and use safe operation specific retry rules. Distinguish retry attempts from unique failed requests . A service can have fewer user visible 5xx while consuming far more paid embedding attempts. Preserve source security and truthfulness: if embeddings cannot be created, do not silently use an outdated cached answer for a different user or return citations from unrelated tenant documents. 5. Beginner friendly numerical production example Capacity: allocated request budget = 6,000 RPM = 100 RPS average . Allocated token budget = 1,200,000 TPM = 20,000 tokens/sec average . Typical input = 300 tokens . Token limited sustainable request rate is 20,000/300 ≈ 66.7 RPS , lower than the RPM derived 100 RPS. At 180 RPS, requested load is 54,000 tokens/sec , or 2.7× the token allowance. Retry multiplication: if each logical call can be attempted three times by SDK × three times by the RAG wrapper , a persistent failure could lead to nine embedding attempts per original request . At 180 originals/sec, the unconstrained worst case upper bound is 1,620 attempts/sec before other client retries. This is an upper bound, not a measured steady state rate, because timing, success, backoff and queueing affect observed attempts. Queue growth: even if the system perfectly throttles to about 66.7 successful 300 token embeddings/sec, new requests arrive at 180/sec. With no shedding, backlog grows at approximately 113.3 requests/sec , or 6,798 additional queued requests in 60 seconds . This approximation ignores burst allowance, failures and varying request sizes; it demonstrates why waiting alone cannot solve sustained excess demand. Admission policy (illustrative): reserve 60 RPS for online use at 300 tokens each (18,000 TPM? Correction: 60×300×60 = 1,080,000 tokens/min , below the 1.2M TPM ceiling) and leave headroom for estimation error/short bursts. A 6,000 RPM limit is not the bottleneck in this allocation; the 1.2M TPM limit is. Retry policy: maximum two total provider attempts per original , a global retry budget no more than 10% of accepted original attempts for a measured interval, and 3 second end to end deadline . These are example engineering choices, not prescribed vendor defaults. Retry After beyond the remaining deadline causes immediate deferral/failure, not an early retry. Steady burst outcome: of 180 incoming originals/sec, suppose 60/sec are admitted , 120/sec are quickly rejected or deferred , and of the 60 admitted 2/sec initially fail transiently . If all 2 retries are authorized under the budget and complete successfully, total provider attempts = 60 initial + 2 retries = 62/sec . That is a 3.3% retry ratio against originals and fits the 10% retry budget. Since retries also consume tokens, request/token reservations must account for 62×300×60 = 1,116,000 TPM , still under 1.2M TPM. Success rate accounting: accepted originals 60/sec can all succeed in this illustrative transient pattern, but only 60/180 = 33.3% of offered requests were served immediately; 66.7% were shed/deferred. Do not misleadingly report “100% success” solely among admitted users. Alternative control: if user facing success rate needs to remain near 100%, provision greater authorized provider quota/capacity or reduce effective demand (validated caching, batching where appropriate, staged launches). Queueing more requests behind a hard 66.7 RPS token ceiling does not sustainably process 180 RPS. After mitigation: a hypothetical measured peak retry amplification declines from up to 9 attempts/original to ≤2 , mean queue age is bounded, and p95 for admitted questions returns to 2.4s while rejected requests fail quickly. Report admitted and offered population latency, shed rate, cost, and grounded answer quality separately. Figures are illustrative , not observed MachinoAI/provider data. The calculation presumes a typical 300 token embedding and simplified average quotas; real quotas, token variability, shared limits and burst windows must be measured. 6. Root cause decision tree 429 coincides with TPM exhaustion while RPM remains available: reduce input tokens, cache identical embeddings, avoid repeated long query strings and tune token based admission, not just request count. 429 coincides with RPM exhaustion but TPM is free: batch eligible requests if supported, increase request budget if authorized, or reduce duplicate calls with normalization/caching. 429 spikes immediately after a deployment or burst below average quotas: check burst/ramp limits, shared provider limits and distributed workers whose local rate limiters are unsynchronized. Coordinate a quota scope controller and ramp gradually. Retry attempts exceed originals × configured max: inspect nested SDK, wrapper, proxy, queue worker and client retries. Confirm attempt counters propagate end to end. Provider 429s continue after incoming originals fall: look for retry backlog, scheduled retries, abandoned queue tasks and orphaned requests that were not canceled. Latency rises before provider errors: measure admission queue depth and age, semaphore waits, gateway backlog, connection exhaustion or worker event loop stalls; rate limiting can be downstream but not the first bottleneck. High 503 and slow down signals: treat as provider overload/ramp behavior per documented error codes and Retry After, not necessarily an account quota cap. Budget exhausted but errors called “rate limits”: distinguish hard spending/account quota from temporary 429. Alert billing/owner rather than automatically resending. Cached answer recovers speed but is wrong/stale: validate source version, requester ACL and grounding separately. An embedding cache avoids computation; a cached answer must obey much stricter freshness/privacy conditions. One tenant causes cross tenant starvation: enforce quota partitioning/fair scheduling rather than each tenant independently assuming the full provider allowance; later curriculum Q281 examines this in depth. Online ingest shares an embedding allocation with offline backfills: reprioritize or defer lower priority indexing; Q233 addresses the separate ingest queue mechanics. A new retrieval source causes tail latency without embedding 429: inspect source fan out/critical path under Q278, not indiscriminately lower embedding concurrency. 7. Engineering solution and rollout A. Trace originals and attempts: one root request ID plus unique attempt IDs across SDK/proxy/orchestrator; record status/error code, request and token budget state, retry count, Retry After, queue wait and cost. B. Inventory effective limits: model/project/org RPM and TPM, provider burst allowances, quota sharing, request/token estimation and contractual capacity. Configuration should be versioned and operator owned. C. Central admission broker: distributed request/token limiter keyed by actual quota scope; bounded online queue, deadline/cancellation checks and in flight concurrency permit; explicit high priority interactive and lower priority batch lanes. D. One retry owner: disable uncontrolled SDK retries, set total attempts and wall clock budget, parse Retry After, use jitter and cap retry load per scope. Avoid nested retries in API clients, worker queues and frontend. E. Controlled rejection: respond quickly with a semantically correct 429/503 and safe retry information or create deferred work only where the product supports it. Never return unsupported RAG content as a “successful” fallback. F. Adaptive recovery: decrease admission on overload signals; ramp back up slowly after sustained successes, avoiding synchronized retry waves. Evaluate total token cost and limit reset behavior. G. Capacity optimization: use authorized caching, input deduplication and carefully selected embedding batch sizes. If offered demand permanently exceeds capacity, purchase/add authorized capacity or reduce demand rather than masking the mismatch. H. Canary test: replay short bursts and sustained load with scripted 429, 503, Retry After values, quota exhaustion, cancellation and multi pod concurrency. Measure user visible acceptance, completion quality, shed fraction, attempts/original and p95/p99. I. Rollback: feature flag admission, retry counts and adaptive rules. Ensure rollback cannot recreate uncontrolled retries, disabled ACL checks or stale document responses. 8. Trade offs and failure scenarios Bounded shedding vs waiting: shedding sacrifices short term acceptance but prevents unbounded tails and protects healthy traffic; waiting may be reasonable for brief bursts if deadlines and queue age stay bounded. High retry allowance: may smooth rare transient 429s but amplifies an overloaded provider; a 10% global retry allowance is an example drawn from SRE practice, not a universal requirement. Very low retry allowance: can prematurely fail a request that would succeed after a short server specified wait; calibrate against measured transient behavior and user deadlines. Static limits: easy to reason about, but may underutilize quota when provider allocations increase or overrun shared quotas when usage changes. Keep a provider informed feedback loop. Adaptive controllers: respond to overload and changing capacity but can oscillate when all API pods update independently. Coordinate by real shared quota scope and use damped adjustments. Token forecasts: uncertainty can cause overadmission; include margin and reconcile actual token accounting. Local per pod token buckets: each pod may admit the entire quota if there is no shared coordination. Avoid accidental N× budget multiplication on autoscaling. Caching: cuts identical query costs, but poor normalization can merge semantically different questions; keep model version and intended text semantics. Batching: saves RPM and overhead but cannot evade token quotas; can worsen interactive tail latency. Provider SDK conveniences: automatic retries may be helpful in isolation but hide multiplied attempts when wrapped by more retrying layers. Backoff without jitter: synchronized clients retry at the same reset time, recreating the stampede. Premature retry vs Retry After: violates temporary limit guidance and increases rejection; if the hinted delay exceeds the request deadline, stop. False green SLO: success rate measured only among accepted requests hides massive shedding. Report offered, accepted, completed, rejected and deferred groups. Failure isolation: embedding overload should not indiscriminately prevent safe nonembedding API routes; keep per dependency budgets and request classifications. Security: do not bypass tenant/RLS or return cached retrieved facts from a different user's permission context under overload. 9. Prevention, alerts and on call runbook Build a golden surge test with configured RPM/TPM, realistic token length distribution, SDK retry configuration, per tenant traffic, ramp spikes, constrained capacity, cancellations and provider response headers. Inject temporary and permanent 429, 503, malformed/missing Retry After, rate limit reset, slow responses, quota scope changes, and callback cancellation. Verify bounded retries and no early retry. Alert on embedding attempts/originals, retry attempt ratio, 429/503 trend, available RPM/TPM, queue age, permit wait, shed fraction, provider cost and fraction of requests outside end to end deadlines. Track offered, admitted, completed, rejected, deferred RPS separately; for admitted requests also track citation validity, retrieval coverage and grounded answer success. Set a safety condition that a canceled/expired user request cannot spawn another embedding attempt , and that known permanent quota/permission errors never retry. For an incident: identify whether RPM, TPM, ramp behavior or spend quota is binding → find first nested retry amplification → disable duplicate retries → cap admission before provider calls → bound waiting → honor Retry After and jitter → coordinate scope wide recovery → check accepted versus offered user outcomes. Rebaseline after provider quota changes, embedding model swaps, new prompt/chunk sizes, new API instances and shifts in online/offline traffic; do not assume quota is unchanged just because application code is unchanged. Review security and privacy for cache keys, embedding logs and test inputs; pressure must not become a reason to leak cross tenant content. Keep release gates on user visible success and honest failure, retry amplification, bounded queues, cost and groundedness—not just reduced 429 counts. Interview takeaway: When embedding 429s create a retry cascade, the fix is rate and token admission control before the provider call, one retry owner, per original and global retry budgets, Retry After aware jitter, deadline and cancellation propagation, and bounded queues/load shedding . Correctly distinguish RPM from TPM and temporary rate limits from hard account quotas. An admission system that safely rejects excess work is healthier than a seemingly successful service that endlessly retries and times out. Research references OpenAI, API rate limits and error mitigation : https://developers.openai.com/api/docs/guides/rate limits OpenAI Cookbook, How to handle rate limits : https://developers.openai.com/cookbook/examples/how to handle rate limits Google SRE, Handling Overload (client side throttling and retry budgets): https://sre.google/sre book/handling overload/ Amazon Builders Library, Timeouts, retries and backoff with jitter : https://aws.amazon.com/builders library/timeouts retries and backoff with jitter/ RFC 6585, 429 Too Many Requests : https://www.rfc editor.org/rfc/rfc6585
Your RAG response latency doubles after adding a third retrieval source. How would you diagnose the critical path and reduce fan-out overhead?
Production incident (illustrative): A SaaS support RAG assistant searches two authorized sources: product manuals (A) and internal runbooks (B). Its full answer p95 is 2.2 seconds. Engineering enables a third source, a connector backed customer knowledge API (C), to increase coverage of niche cases. Full answer p95 rises to 4.4 seconds, even for questions answerable entirely from A. A waterfall reveals A completes in 170 ms, B in 240 ms, and C usually completes in 320 ms but sometimes stalls for 2,100 ms. The orchestrator starts all three in parallel but waits for every response , then globally reranks 75 instead of 30 candidates and expands prompt tokens. “Parallel search” did not mean “constant latency”; both the slowest branch and downstream merge/rerank/generation work increased. Core principle: Identify the end to end request critical path and separately account for (1) additional required fan out dependencies, (2) connector tail latency, (3) selection/merge/rerank overhead, (4) context token inflation and LLM output latency, and (5) concurrency, retries, connection pools and queuing. Optimize information gain per unit of critical path latency , never source count alone. Research on The Tail at Scale shows that as a request waits for more dependent services, rare slow operations can dominate aggregate responsiveness; Microsoft’s RAG guidance describes parallel multi source retrieval but notes that reranking and expanded context introduce additional latency/cost. 1. Establish an apples to apples regression Freeze a representative set of authorized question/answer tasks, query rewrite, model revisions, source document versions, tenant/ACL, top k, prompt template, output token cap, hardware, traffic level and cache hit ratios. Compare A+B with A+B+C via paired replay and a feature flag. Segment by routing intent: (i) questions solvable using A/B, (ii) questions requiring C, (iii) mixed source comparisons and (iv) ambiguous queries. A source that improves one segment may be an avoidable dependency in another. Define metrics for full response p50/p95/p99 , streaming time to first token (TTFT), answer correctness/grounding, per source recall, unique evidence contributions, source routing accuracy, cost, token use and failure/timeout fraction. Do not assume doubled p95 implies one source doubled its average duration. Capture trace timings at the same traffic/concurrency and compare paired or stratified percentile distributions , with confidence intervals and provider/source status codes. Determine whether observed “latency” is wall clock until all retrieval completes, until first generated token, or until last streamed token. A faster retrieval span may not improve the user visible completion time if generation dominates. Keep source C’s permissions intact in all tests. It is unacceptable to route around mandatory access controlled evidence merely to achieve a latency target. 2. Instrument a per request critical path trace Parent spans: rag.request → query.prepare → retrieval.fanout (child A, B, C) → merge.dedupe → rerank → context.pack → llm.generate → response.render. Record start ns, end ns, queue wait ms, service ms, source id, candidate count, input tokens, output tokens, cache outcome, retries and content version hash. Annotate dependency edges , not just spans: rerank requires which sources? Does llm.generate start after all three, after a quorum, or after a sufficiency gate? Critical path calculation requires knowing what blocks the next step. For a true parallel all required fan out, retrieval wall clock is approximately max(T A,T B,T C)+orchestration, not T A+T B+T C. If the implementation accidentally performs sequential await calls before launching the next fetch, its wall clock is closer to a sum. For p95 diagnosis, avoid p95(A)+p95(B)+p95(C) or even simply max(p95(A),p95(B),p95(C)) as an exact request p95: per request correlation and tail dependencies matter. Derive end to end p95 from complete recorded requests. Separate connection pool and semaphore queue wait from actual remote service duration. Adding source C can exhaust the shared HTTP client pool, increase DNS/TLS connection creation or block A/B even if C is not itself on every query's critical path. Trace 429/503, retry count, redirect/auth refresh, connector rate limits, circuit status and cancellation. Nested retries can make C appear to have an inexplicable 2 second median. Capture merge/dedup latency, fusion branches, global reranker candidates and token budget utilization. Source C may contribute no useful answer evidence while forcing many more candidates through a costly cross encoder. Correlate p95/p99 with cache misses, source C request count, large C payloads, fan out width, tenant, region and prompt length. If only some queries regress, it is likely an eligibility/routing or tail problem rather than universal LLM slowness. 3. Locate the first regression boundary A. Query planning: inspect source selection traces. Did the router start calling C on every question despite C containing only specialized content? Check routing false positives and whether the source is optional or mandatory for the task. B. Fan out start: compare actual A, B and C start times. If calls are launched serially, fix concurrent scheduling within bounded per source limits; do not sequentially await A before launching B and C. C. Synchronization: identify an all of barrier, gather, or framework node that waits for slow C, even when adequate A/B evidence already exists. A “parallel” implementation can still have tail latency equal to the slowest required source. D. Infrastructure contention: measure HTTP connection pool queue, provider specific semaphore, container worker count, CPU, memory, network and egress. If A/B got slower after C was enabled, C may be starving shared transport resources. E. Candidate explosion: if global candidates grow from 30 to 75, profile dedup, fusion and cross encoder rerank. Guard against near duplicate C passages crowding A/B evidence and increasing context tokens. F. Generation inflation: compare exact final prompt byte/token counts, truncation, model version, LLM time to first token and output length. Increasing evidence can double LLM prefill or token charges depending on model, even when retrieval remains fast. G. Cache behavior: check whether fan out bypasses a previous A/B cache key or causes cold C lookups on every request; inspect cache invalidation scope and source version safety. H. Concurrency: use stepped load to see whether C saturates its own quota and triggers retries or pushes shared orchestrator concurrency over an admission limit. Q276 covers general capacity; Q277 covers embedding retry storms. I. Evidence value: compare recall@k, per question C only gold span coverage and correctness with/without C. If C supplies indispensable citations in 12% of questions, unconditional deletion is unsafe; optimize routing. J. Regression attribution: use controlled A/B and A/B/C replay with exact evidence manifests, identical answer model settings and source version pins. A change in grounding, citation support or protected exception coverage is part of the regression. 4. Controlled experiments that isolate overhead Experiment 1: three way waterfall —record 1,000 paired requests before/after source C (same query mix and arrival pattern), with per span wait/service times and answer quality labels. Experiment 2: C disabled versus enabled —leave the reranker, context packer and generator configuration identical. If retrieval span increases sharply but generation token usage is constant, look at C/barrier/connection pools. Experiment 3: C stubbed —return a fixed, authorized canned C response at negligible latency in an isolated test. If p95 remains high, candidate processing and generation are likely major causes; if it falls, C transport/server/tail is implicated. Experiment 4: fixed fan out, fixed candidates —feed a frozen A+B+C result set to fusion/reranking and compare 30 vs 75 candidate sizes; this isolates selection overhead from remote waits. Experiment 5: identical packed prompt —replay a fixed serialized LLM input, regardless of source count. A model delay that disappears under prompt pinning implicates extra tokens/content or provider load rather than “number of retrieval sources.” Experiment 6: selective routing —send A/B sufficient questions to A+B only, and C dependent questions to A+B+C. Measure router false negatives that could hide critical evidence. Experiment 7: tail injection —simulate a 2 second C delay and test wait all versus an authorized, sufficiency checked C skipping policy. If C is required for the user’s answer, no latency optimization may silently drop it. Experiment 8: controlled concurrency —increase arrival rates and compare queue wait/connection usage to separate source C’s intrinsic slow calls from orchestration resource saturation. Use randomized interleaving/counterbalanced order for offline evaluation to avoid model/index drift, plus paired bootstrap over query IDs; monitor errors and source freshness, not only latency. 5. Beginner friendly numerical production example Before adding C, each request has 150 ms embedding/query preparation, parallel retrieval A 170 ms and B 240 ms , 110 ms merge+rerank, 100 ms context packing, and 1,400 ms generation. Approximate full critical path: 150 + max(170,240) + 110 + 100 + 1400 = 2,000 ms (2.0 seconds). After adding C, representative slow path requests have A 170 ms , B 240 ms , C 1,250 ms (including connector queuing). Merge+rerank rises to 310 ms because candidates grow 30 → 75 ; context packing increases to 160 ms ; generation increases to 1,950 ms because the prompt grows. Critical path: 150 + max(170,240,1250) + 310 + 160 + 1950 = 3,820 ms (3.82 seconds). A mild additional cold start or request/response delay of 180 ms can produce 4.0 seconds , approximately double the previous 2.0 seconds. Attribution for the 1,820 ms core difference: C changes fan out barrier from 240 → 1,250 ms ( +1,010 ms ); reranking +200 ms ; packing +60 ms ; generation +550 ms . Thus the largest contributor is the C dependent barrier, but downstream work accounts for 810 ms , so optimizing C alone cannot restore the 2 second baseline. Selective source value: suppose 1,000 labelled queries , of which 850 can be correctly answered with A/B and 150 require C for an authoritative fact. A router identifies 140/150 required C queries (93.3% recall) while unnecessarily sending 90/850 A/B only queries to C. Total C calls: 140+90=230 instead of 1,000, a 77% reduction in C call volume, but 10 C required questions are missed and must be caught through uncertainty/refetch or improved router recall before rollout. Optimization: query aware routing + capped per source candidate depth + dedup before cross encoder rerank yield an illustrative 2.35 second p95 over a defined mixed workload, compared with 4.4 seconds after unconditional C. That's a 46.6% p95 reduction , assuming source coverage/groundedness remains acceptable; these p95s are separately measured aggregate hypotheticals , not sums of per stage p95s. Per source tail risk: if C has independent p95 of 1,250 ms while A/B are ~200 ms, each mandatory C request still needs a realistic C specific latency/quality trade off. A source aware deadline and partial result contract are explored under 279, and strict cross stage deadline allocation under 280. Counterexample: simply removing C restores fast answers but misses 150/1,000 = 15% of questions. That improvement in latency would be unacceptable if those questions are critical. Track C only evidence recall, not only average p95. All quantities are illustrative , not observed MachinoAI timings. The deterministic sums depict individual hypothetical request paths, not percentile arithmetic; real p95 values require many actual traces. 6. Engineering solutions and rollout Query aware source routing: predict candidate source(s) from intent, entity/product, region and authoritative scope. Make C optional for A/B sufficient questions, but require it for known C dependent categories; calibrate false negatives against high severity evidence. Parallel but bounded fan out: launch independent calls concurrently, with separate per source concurrency limits and isolated HTTP connection pools where justified. Bound total calls/query, reduce duplicate decomposed searches, and share reused embeddings. Evidence sufficiency gate: after A/B results arrive, test whether they cover each required answer facet from an authoritative source. Only skip C when the result is sufficient and policy permits source omission; if uncertain, fetch C or disclose limited evidence. Two phase retrieval: run fast source(s) first for common queries, then selectively invoke slower C on uncertain/complex cases. This saves calls but can increase C required latency because C no longer starts in parallel. Compare staged versus eager parallel routing on source dependent segments. Source aware candidate budgeting: cap and prioritize C candidates by relevance, authority and unique fact contribution rather than blindly appending its top k. Dedup before expensive reranking; retain required exceptions and multi source coverage. Reranker capacity controls: batch efficiently and cap cross encoder comparisons; use source specific score calibration for fusion, not raw score comparisons across heterogeneous indices. Early, safe decisions: in low risk cases, a verified A/B answer may be generated without C, while a mandatory C case must wait or honestly report uncertainty. Never claim complete evidence after intentionally dropping an authoritative dependency. Transport optimization: reuse connections, avoid repeated authentication handshakes, cache safe metadata and honor source quotas; isolate pathological connector retries. Store the right document/version and tenant ACL in cache keys. Streaming and background workflows: streaming an initial token earlier can improve perceived response time but should not present an incomplete high risk decision as final. For optional supplemental content, support clear post answer follow up only if product semantics permit it. Deadlines and cancellation: propagate absolute request deadlines, cancel unneeded optional branches, and avoid wasted work after client disconnect; Q279 goes deeper into bounded partial results and external API deadlines. Canary and rollback: ship source router, concurrency/fan out limits, candidate budgets and sufficiency checks behind independent flags. Shadow evaluate source coverage and p95 on multi tenant samples, then canary without bypassing authorization/source currency. 7. Trade offs, pitfalls and common mistakes More sources ≠ slower serial work: properly parallel source calls overlap, but the slowest required branch still determines barrier completion. More sources also raise probability of hitting a slow tail. Source routing: cuts unnecessary calls but can suppress rare, critical evidence. Use explicit high risk routing overrides, query level coverage audits and fallback on uncertainty. Early finish: can reduce tail latency but may return incomplete or misleading answers if C has mandatory facts. Ground the sufficiency test in a verified requirements inventory. Global candidate cap: improves compute but may disproportionately drop niche source evidence. Use diversified per source/facet quotas and measure C only recall. Cache: speeds repeated access but may leak cross tenant content or serve expired policies when version/permission keys are weak. Verify authorization at retrieval/render time. Hedged duplicate requests: can reduce rare tails but raises downstream load and cost; never blindly hedge a rate limited C. Use only under measured spare capacity and per request budgets. Sequential source search: may avoid unnecessary work when A/B suffice, but makes true C dependent requests slower; test task conditioned routing. Reranker expansion: bringing 75 candidates into an expensive cross encoder without dedup can consume more latency than the new retrieval source itself. Parallelism without isolation: C can exhaust the shared HTTP connector pool and worsen A/B; monitor queue wait per pool. Percentile math: adding stage p95s or averaging per source p95s does not produce overall p95. Analyze timestamped per request traces and critical path dependencies. Prompt inflation: extra source text can increase generation prefill, monetary cost and hallucinations. Preserve sufficient authoritative context rather than blindly concatenate. Overoptimizing TTFT: a quick first token does not substitute for full answer freshness, completeness and citation support. False improvements: benchmark not only speed but source specific answerability, unsupported citations and tail errors. A fast incorrect answer is a failed RAG request. 8. Prevention and production runbook Define a per source contract: eligibility/routing rules, authorization, expected latency distribution, response size, rate limit, cost, source freshness, cache policy, failure behavior and mandatory versus optional semantics. Regression test adding any new source using representative answerable by existing versus uniquely answerable by new source queries; measure p95/p99, TTFT, source contribution, candidate/LLM token inflation and wrong source citations. Require distributed traces of every fan out branch and downstream merge/rerank; alert on rising fanout width, C call fraction, source tail, shared pool wait, cross encoder candidate count and final prompt tokens. Maintain golden queries requiring multiple sources and critical exceptions. A router that avoids C on ordinary questions but fails these cases must not pass rollout. Inject a delayed C dependency, retry storms, cache misses, connection pool exhaustion, wrong source routing and oversized C candidate responses; verify failure isolation and safe fallbacks. For an incident: compare before/after paired traces → inspect fan out dependencies → identify max branch/barrier → measure queue wait vs remote service → inspect candidate count and prompt tokens → run C stub/fixed candidate tests → test selective routing → recheck coverage and latency on the full cohort. Gate rollout on both latency (p95/p99, TTFT) and quality (C only gold span recall, evidence completeness, citation validity, no unauthorized leakage), as well as per request token/connector cost. Keep source C feature flag and a conservative router rollback; fail closed for queries whose authoritative evidence cannot be fetched instead of silently substituting unrelated sources. Interview takeaway: When a third RAG source doubles latency, find which dependencies are on the critical path , not just which span is slow. In parallel fan out, the slowest required source controls the join barrier; candidate merge/rerank and extra prompt tokens add more delay. Use distributed traces and source ablation/fixed candidate experiments, then reduce unnecessary calls through query aware routing, evidence sufficiency checks, bounded fan out and candidate budgets while preserving unique source recall. Research references Dean and Barroso, The Tail at Scale , Communications of the ACM 2013: https://research.google/pubs/the tail at scale/ Google SRE, Monitoring Distributed Systems—Tail Latency : https://sre.google/sre book/monitoring distributed systems/ Microsoft Azure Architecture Center, RAG Information Retrieval Phase : https://learn.microsoft.com/en us/azure/architecture/ai ml/guide/rag/rag information retrieval Microsoft Azure AI Search, RAG and AI Search : https://learn.microsoft.com/en us/azure/search/retrieval augmented generation overview Microsoft Azure Architecture Center, Agentic RAG Design : https://learn.microsoft.com/en us/azure/architecture/ai ml/guide/rag/rag agentic