A RAG assistant gives different answers to the same question across repeated requests. How would you separate retrieval nondeterminism from generation nondeterminism?

Production incident (illustrative): A procurement RAG assistant answers the *same* question—“Who must authorize Singapore contractor lodging exceptions in 2026?”—three ways across five requests: “Regional Finance Director,” “line manager,” and “finance team.” All requests return HTTP 200. One trace cites the current contractor exception policy; another cites an older employee travel FAQ; a third cites the correct policy but paraphrases its approval condition incorrectly. Changing temperature to zero does not eliminate all differences. The team must identify whether the evidence changed, the final model input changed, or the model interpreted identical evidence differently.

Core principle: Diagnose repeat-answer variability with a three-boundary replay: (1) hold the natural-language request fixed and inspect retrieval; (2) hold the retrieved candidates fixed and inspect reranking/context assembly; (3) hold the exact serialized model input fixed and inspect generation. Separate lexical variation (different wording, same supported decision) from semantic/grounding instability (different approvals, amounts, dates, exceptions or citations). Zero temperature and a seed can improve repeatability but are not absolute guarantees across all model/providers and deployments.

1. Contain the incident and define “same request”

• Verify exact user question, conversation history, user/tenant identity, ACL, locale/timezone, query timestamp, feature flags, model routing, source snapshot, and effective-date context. A request containing “today,” an updated chat memory, or changed user permission is not the same semantic task even if the last user message is identical.
• For decisions involving approval authority, policy eligibility or money, avoid selecting the most frequent answer by majority vote if cited evidence conflicts. Require current authorized source support; abstain or clarify when the controlling rule is unavailable.
• Capture request and stage correlation IDs: raw-input hash, normalized/rewrite text, query vector hash, embedding model/revision, source alias/index generation, filter JSON, ANN parameters, selected shard/replica, dense/BM25 candidates with ranks, fusion/rerank output, packed context, final serialized prompt hash, model name/revision, sampling settings, seed if supported, response fingerprint if supplied, output text and citations.
• Keep trace payloads access-controlled; hashes and structured candidate IDs often suffice for production telemetry. Securely retain exact text only for approved diagnostic replay. The test harness must respect source ACLs and deletion/retention policy.
• Label the correct answer from a current authoritative passage with source/version/offset. Set task-specific invariants: correct approver, approval-before-booking, applicable jurisdiction, no employee-policy substitution and properly supported citations.

2. Classify variability before debugging

• Style-only: “Director of Regional Finance” vs “Regional Finance Director” can be equivalent if both cite the same valid clause; exact string mismatch is not necessarily a defect.
• Evidence variation: candidate IDs, content hashes, source versions or scope filters change. First investigate retrieval, reranking, context assembly, source freshness or access control.
• Generation variation: the *exact same serialized prompt* and model settings yield different decisions, unsupported claims, omission of exceptions, or citations. Inspect decoding/model versions and grounding behavior.
• Hybrid/multistage variance: retrieval is stable, but dynamic query routing, LLM query rewriting, reranking or packing changes the prompt. This is neither first-stage vector instability nor pure decoder randomness.
• Temporal/version change: documents legitimately update during the test. Different answers can be correct for different as_of timestamps; compare only when the authorized source snapshot and requested effective date are controlled.
• Failure severity: a changed adjective is less severe than a changed numeric threshold, policy approver, effective date or unauthorized citation. Measure critical decision consistency separately from text similarity.

3. The four-cell controlled replay matrix

• A. End-to-end repeated requests: issue 20–50 identical, authorized test requests under a pinned session/tenant/source snapshot. Record all stage outputs. This measures observed variation but does not yet attribute cause.
• B. Frozen input + retrieval-only: generate the embedding once, pin the literal query vector, filters, index alias/generation, ANN settings and ideally a replica. Run vector and lexical search repeatedly without calling the answer model. Compare top-k source IDs, content hashes, authorized relevant spans and exact/ANN reference results.
• C. Frozen retrieval + assembly-only: inject a fixed, ordered list of authorized candidate objects and deterministic reranker inputs into the context builder; rerun fusion, deduplication, compression, sorting, token packing and citation serialization. Compare exact final prompt hashes and included source offsets. If these change while input candidates are fixed, context construction is unstable.
• D. Frozen exact prompt + generation-only: replay precisely the same serialized system/developer/user messages, tool outputs, evidence spans and model settings directly to the generator repeatedly. Compare answer decisions and citations with a domain-aware evaluator. Changing output with an identical prompt indicates generation/backend nondeterminism, not vector retrieval.
• E. Cross-factor intervention: replay multiple fixed candidate lists with fixed prompt templates across matched model revisions. Use randomized, balanced assignments if evaluating whether each retrieval variant causes a response change; do not compare sequential samples after an unrecorded model release.
• F. Hold timestamps and caches constant: pin source content and policy-effective date for a diagnostic run. Record cache keys; two requests with the same query text but different selected source generations or cache entries are not equal inputs.
• G. Independent reference: if ANN results vary, compare exact/flat nearest neighbors on the *same filtered corpus*; if the final prompt varies, compare its token/byte-level diff; if generation varies, check model/provider fingerprints, sampling and actual response content.

4. Retrieval-side root-cause debugging

• Embedding/preprocessing: capture query rewriting, spelling normalization, chat-history condensation, query embedding inputs and the resulting vector fingerprint. An LLM rewriter can produce different search queries even when the user question is unchanged; pin its result when testing search itself.
• Approximate search: HNSW and other ANN methods can change the rank order of nearby candidates, especially around top-k boundaries. Qdrant documents slight ranking shifts and recommends exact search when stable ordering is required. Determine whether the changed neighbors affect *actual evidence coverage*.
• Replica consistency: a current policy can be visible on one replica and not another during updates. Qdrant documents configurable read consistency and read affinity; investigate peer-specific source versions, optimizer readiness and deletion propagation. Affinity may hide rather than repair stale state.
• Filters/time-dependent policies: verify tenant, role, region, status, source version and effective date are serialized identically. Avoid relaxing ACLs to make results consistent. A current and an expired source are not interchangeable.
• Fusion and reranking: compare branch depths, RRF/rank fusion, score normalization, candidate deduplication, stochastic LLM rerankers and tie-breaking. If first-stage candidates match but their final order changes, label the issue “selection/assembly,” not vector retrieval.
• Concurrency: production load, partial-shard responses, timeouts, retries, caches and background ingestion can change candidate completeness. Distinguish a successful complete response from a partial result treated as success.
• Source changes: pin document/chunk IDs, content/version hashes and index alias. Identical-looking citations may point to altered content if the resolver follows a mutable latest-version URL.

5. Generation-side root-cause debugging

• Exact prompt equality: assert stable serialized messages, source spans, instruction order, context section order and tool content. A stable “question” field alone does not prove identical model input.
• Sampling parameters: record temperature (if supported), top-p/top-k, max output tokens, stop sequences and seed. Provider support differs by model; OpenAI documents best-effort reproducibility with fixed seed plus system_fingerprint on applicable APIs, while even temperature zero is not a universal determinism promise.
• Backend/model drift: compare immutable model snapshot where available, provider version, system_fingerprint if provided, inference region, safety settings and tool-use mode. A model alias can move to a new revision even when client config stays the same.
• Evidence interpretation: same current policy can generate different approval outcomes because of negation, exception omission, ambiguous pronouns, prompt length, position sensitivity or conflicting older policies. Evaluate *claim support* and *qualified decision correctness*, not only string consistency.
• Output formatting/citations: test whether source IDs are validated by a deterministic resolver or generated as free text. Different URLs/citation labels can be a citation-rendering defect, separate from decision content.
• Length and termination: output token caps or stop conditions may truncate an exception in only some runs, causing apparent semantic disagreement. Record finish reason and generation tokens.
• Independent reviewer: judge semantic equivalence and citation support against a verified source. An LLM-as-judge has its own error rate; human-audit critical disagreements.

6. Worked numerical production example

• Take 100 distinct expert-labelled queries, each run 10 times (1,000 responses) under the same authorized policy snapshot. Define a stable decision as the same correct, source-supported approval outcome across all ten attempts, not identical wording.
• Baseline: for 30 queries, the retrieved authoritative source set changes across repetitions; 20 of these exhibit a changed final decision. For 70 queries, retrieval and final prompt stay byte-identical; 18 still exhibit changed decisions. The other 62 queries remain decision-stable. Thus 38/100 = 38% of query groups show material answer variability, but one cannot assign all 38 to retrieval: 20 are retrieval-associated; 18 occur with fixed prompts and demonstrate generation-side variation.
• One affected query has source lists R1=[CurrentPolicy, ExceptionMemo, FAQ] and R2=[OldEmployeeFAQ, CurrentPolicy, FAQ]. Its citation-set Jaccard is 2/4=0.5 when distinct source IDs are used. The correct authoritative exception survives in R1 but not in the final packed prompt for R2. This path is retrieval/selection-associated, not proof that ANN itself failed.
• Controlled retrieval-only replay pins the vector and filter, revealing a source-version mismatch on two replicas. Correcting replica freshness and stable source routing resolves 16 of the 20 retrieval-associated variable decisions, leaving 4 whose candidate/packing changes require further investigation.
• Frozen exact-prompt generation replay reproduces decision differences for 18 query groups. Fix an ambiguous instruction, add span-grounded approval verification and use supported low-variance decoding settings. Suppose this eliminates variability for 14 of 18, leaving 4 generation-sensitive groups for further human review.
• On the same 100-query benchmark, assume there is no overlap between the 20 retrieval-associated and 18 frozen-prompt groups as defined in this controlled example. The remaining variable groups total 4+4=8/100 = 8%, down from 38%, a 30-percentage-point reduction. In real deployments, interactions/overlap mean you must remeasure end-to-end variance rather than naively add reductions.
• Suppose p95 rises from 150 ms to 205 ms due to stronger consistency, extra provenance checks and verification. This is +55 ms. These numbers are illustrative; measure latency, freshness, availability, grounded accuracy and cost before rollout.
• An unchanged answer that is consistently wrong is not success. The evaluator must also check accuracy and citations: deterministically retrieving an expired employee policy would reduce variance while increasing harm.

7. Engineering solution and safe rollout

• Versioned trace contract: correlate query_hash, rewritten_query_hash, embedding_hash, retrieval_manifest_hash, packed_prompt_hash, answer_claim_hash, source versions, ACL snapshot, planner/model revisions and replica identity across requests.
• Stage-specific reproducibility tests: on every release, rerun (i) fixed-vector retrieval, (ii) fixed-candidate context packing, and (iii) frozen-prompt generation. Each should report variation independent of other stages.
• Canonicalize for metrics: use stable source/version/section IDs to compare retrieval sets; for answers, compare structured decision fields (approver, condition, amount, date), normalized claims and citation support instead of punctuation or stylistic text.
• Reduce source variance: pin embedding and index generations, inspect replica consistency, benchmark ANN candidate depth against exact search, and detect missing/deleted source versions. Stronger consistency has latency and throughput costs.
• Stabilize selection: deterministic sorting with documented tie-breaks where applicable, stable fusion windows, version-aware deduplication and a consistent context budget. Where LLM query rewrite/rerank is used, capture and reuse its outputs for replay.
• Improve grounded generation: require current authorized spans for each decision, preserve exception qualifications, verify claims after generation and abstain when evidence is ambiguous. Low temperature alone does not fix missing or conflicting evidence.
• Provider-aware reproducibility: on an API supporting seeds, hold seed and all settings fixed and record backend fingerprints. Treat exact output matching as best-effort; benchmark on task-correctness and citations, not a promise of byte-for-byte generation.
• Shadow then canary: roll out corrected retrieval settings and generation controls independently where possible. Attribute changes to each intervention using controlled cohorts, and monitor variance and accuracy together.
• Safeguarded rollback: roll back config/model versions without reviving deleted or unauthorized documents. Retain current ACL and policy validity checks regardless of the candidate source index.

8. Trade-offs, common mistakes and limitations

• Temperature zero: usually reduces sampling randomness when supported; it does not freeze prompt assembly, backend model revisions, GPU execution or ambiguous reasoning.
• Fixed seeds: useful in supported APIs but provider/model dependent and may still differ after system changes; never claim guaranteed determinism.
• Exact search: provides a more stable diagnostic reference than ANN but costs more latency/compute for large corpora. Use selective exact checks for critical query slices.
• Read consistency/affinity: can reduce replica-dependent blinking, but stronger reads add load and fixed affinity can conceal a lagging replica. Verify deletion/ACL propagation.
• Caching responses: makes repeated strings identical but can lock in wrong answers, stale documents or permissions; key and invalidate by tenant, source version, ACL policy and effective date.
• Majority vote across generations: can produce a confident consistently wrong answer if all generations interpret one stale policy incorrectly. Prefer verifiable authoritative evidence over consensus.
• Prompt simplification: reduces some variation but may remove qualifiers and increase omission errors; preserve source and citation conditions.
• Repeated-run cost: 10–50 repeats per query can be expensive; use a stratified golden set, adaptive sampling and human review for high-impact disagreements.
• Evaluation granularity: overlap@k and embedding similarity can show candidate movement but do not prove harmful answer drift. Measure actual decision invariants and supported citations.
• Stochastic infrastructure: retries, fallbacks, load balancing and indexed-data freshness can change the complete retrieval graph; trace them instead of assuming “same API call = same inputs.”

9. Prevention checklist and production runbook

• Maintain a golden set with policy exceptions, current versus expired documents, equivalent phrasings, fixed-number approvals, ambiguous citations and cases whose relevant source is near the top-k boundary.
• Run fixed-query repeated end-to-end, fixed-vector retrieval-only, fixed-candidate assembly-only and frozen-prompt generation-only suites after changes to model, indexing, rerank, prompt, tokenizer, deployment or source routing.
• Alert separately on retrieval candidate variance, final prompt hash variance, material decision variance, unsupported claim rate, wrong-policy citation rate, stale replica reads and p95/p99.
• Log exact source/version and effective-date applicability for every decision; check ACL and current deletion status before answering even when retrieval is cached.
• Use human adjudication for semantic disagreements; measure pairwise both-correct rate, repeated-run all-correct decision rate, and proportion of responses with consistent but incorrect answers.
• Inject faults: one stale replica, inconsistent index alias, changed query rewriter output, unstable fusion tie-break, dynamic chat history, model rollout and a changed provider fingerprint.
• Set clear rollback triggers based on grounded decision correctness, not merely response consistency. A stable hallucination is still a failure.

Interview takeaway: Repeated RAG answers can vary because retrieval returned different evidence, context assembly produced different prompts, or generation changed despite identical prompts. Freeze and replay each boundary independently—fixed vector, fixed candidate list, and exact serialized prompt—then measure material decision and citation consistency alongside factual correctness. Adjust the first failing stage rather than blindly setting temperature to zero.

Research references

• OpenAI, Advanced usage: reproducible outputs, seed and system fingerprint: https://developers.openai.com/api/docs/guides/advanced-usage
• OpenAI Cookbook, Reproducible outputs with seed: https://developers.openai.com/cookbook/examples/reproducible_outputs_with_the_seed_parameter
• Qdrant, Search and ranking stability FAQ: https://qdrant.tech/documentation/faq/qdrant-fundamentals/
• Qdrant, Consistency guarantees and read affinity: https://qdrant.tech/documentation/scaling/consistency-guarantees/
• Redis, RAG debugging and failure-stage diagnosis (2026): https://redis.io/blog/rag-debugging-guide-retrieval-errors/