A user asks a multi-part question, and the answer addresses only the first part. How would you determine whether retrieval coverage or generation is responsible?

Production incident (illustrative): An enterprise procurement assistant is asked: “What is the Singapore contractor hotel cap, who approves an exception, and what evidence must be attached to a reimbursement claim?” The answer states “SGD 350 per night”, cites the current contractor policy, and stops. It neither names the Regional Finance Director who authorizes a documented exception before booking nor explains the required written approval and supporting receipt. The cap is factually correct, and the citation is genuine, but the user requested three distinct outputs. The team cannot tell whether source retrieval failed to cover the approval/documentation clauses or whether the LLM ignored those clauses in its prompt.

Core principle: Treat every meaningful subquestion as a traceable coverage obligation from user intent through retrieval, reranking, context assembly, generation, and displayed response. Ask *where the first missing subquestion occurred*. If evidence for part 2 was never retrieved, generation cannot legitimately answer it; if it was in the exact final model-visible prompt, then the omission is a generation, answer-budget or post-processing issue. Microsoft Azure RAG guidance describes query decomposition for complex questions and separately evaluates response completeness—whether an answer resolves all aspects of a user's request.

1. Establish the question contract before changing retrieval

• Freeze the exact user question, conversation turns, region, entity, tenant/ACL, policy-effective date, model, retrieval config and corpus snapshot. Clarify whether the three requested parts are independent or whether one depends on another.
• Decompose without answering into atomic intent slots. For this case: P1 = standard hotel cap, P2 = exception approver and timing, P3 = supporting documentation for reimbursement. Retain the original wording and referents for each part; the third part concerns *reimbursement evidence*, not generic onboarding paperwork.
• Make each subquestion self-contained using authorized contextual identifiers: “For a Singapore contractor under the effective 2026 travel policy, what is the nightly accommodation cap?”; “Who can authorize a cap exception, and when?”; “What attachments are necessary for reimbursement?”
• Capture a reference answer checklist from current authoritative sources: P1: SGD350/night; P2: prior written Regional Finance Director approval when applicable; P3: receipt and written exception approval where relevant. These policy details and figures are illustrative; validate actual enterprise policy before implementation.
• For each slot, store slot_id, intent, expected_fact_set, source_scope, source_span_ids, applicability, severity, dependencies, answerability. Preserve whether different policy sources or tools are required.
• Do not blindly split every query into many pieces. Single-fact questions may be handled more efficiently by one search; multipart decomposition is justified when independent facts or evidence sources are likely.
• Establish that all requested parts are authorized to answer. If a part requires restricted employee data, absence of that unauthorized detail is not a retrieval-completeness failure.

2. Build an evidence-survival matrix per subquestion

• Source availability: verify an authorized, active source actually contains authoritative evidence for each slot, using SME-labelled document/version/span identifiers.
• Planning coverage: check whether each slot received a query plan or was accidentally merged into a weaker “hotel cap” search. Log all planner outputs, source routes and reasons for omitting a subquery.
• Retrieved-candidate coverage: measure which slots are supported anywhere in raw dense/lexical results before fusion. An answerable part missing here is a search/planner/source problem.
• Reranked coverage: inspect candidate selection and fusion. The right document may have been retrieved but an exception clause could be pruned by top-k, source grouping or duplicate suppression.
• Final-context coverage: inspect the exact serialized prompt sent to the reader, including citation IDs and source offsets. Verify all relevant snippets survived compression, token budgeting, metadata serialization and final prompt ordering.
• Generator obligation coverage: extract answers at the granularity of P1, P2 and P3 from the model's full raw output, not merely visible summary text. Measure correct, grounded completion or explicit justified abstention for each.
• Displayed-response coverage: inspect response post-processing, length caps, streaming termination, UI summarizers and citation rendering; text may be generated correctly and then hidden or truncated.
• Assign a first-loss reason code per slot: SOURCE_ABSENT, PLAN_OMITTED, RETRIEVAL_MISS, RERANK_DROPPED, CONTEXT_TRIMMED, GENERATION_OMITTED, OUTPUT_TRUNCATED, UNANSWERABLE or UNAUTHORIZED. One request may have a different cause for each slot.

3. Controlled experiments to separate retrieval and generation

• Experiment A — query decomposition audit: hold the corpus/index fixed. Compare an original composite query against individually searched P1, P2 and P3, using the same filters, source authorization and candidate budgets. If P2/P3 succeed individually but not in the composite plan, the problem is planner/query dilution, not missing indexed content.
• Experiment B — oracle evidence injection: bypass real retrieval for one controlled test and supply exact authorized gold spans for all three slots to the original answer model, preserving source metadata. If the model still answers only P1, generation/prompt instructions/response length are implicated.
• Experiment C — frozen final prompt replay: hold identical serialized system/user messages and the same P1/P2/P3 evidence, generation settings and model revision; repeat generation. If P2/P3 are inconsistently omitted, inspect decoder variation and prompt overload. If always omitted, inspect instructions, task decomposition and output schema.
• Experiment D — fixed generator with retrieved prompts: replay actual candidate lists and their assembled contexts through the same model with the same settings. Compare whether missing slots track missing evidence or persist when the evidence is present.
• Experiment E — expanded retrieval only: improve coverage for P2 and P3 using targeted subqueries but do not change the generation prompt. If final completeness improves when new evidence is included, retrieval was a limiting factor—though generation can remain independently imperfect.
• Experiment F — generation only: keep the *same complete, authorized evidence set* and add explicit answer slots/coverage validation. If completeness improves without retrieving anything new, generation or answer validation was a limiting factor.
• Experiment G — interactions: test both targeted retrieval and structured generation in a paired 2×2 evaluation; source availability, position effects and token limits may interact. An observed gain from retrieval does not prove the decoder was perfect.
• Safety: never manufacture oracle evidence in production. Oracle injections are labelled *offline tests* using permitted, verified source spans; no evaluation result should be interpreted as a production source.

4. Root-cause decision tree with production examples

• P2 never planned: intent parser interpreted the complete question as a single hotel-cap lookup. Fix decomposition, check that every explicit conjunction (“and,” “also,” “plus”) yields appropriate slots, and run distinct subqueries.
• P2 planned, no matching source candidates: search rewrite, source routing, wrong tenant/filter, embedding dilution or stale index. Validate per-slot exact/lexical retrieval and authorized source completeness.
• P2 candidate found, discarded in fusion/rerank: diagnose a common policy overview dominating the top-N and a specific approval clause ranking lower. Apply per-slot evidence coverage constraints or source/facet-aware selection.
• P2 passage retrieved and selected but absent from serialized prompt: context trimming, dedup or compression. Repair span survival as in Q266, with source offsets and token-budget tests.
• P2 present in final prompt, answer still omits it: model neglects the second intent, adheres too strongly to “answer in one sentence,” suffers from context position/noise, or makes an incomplete synthesis. Test slot-based generation and Q267-style position controls.
• P2 correctly expressed, P3 missing: investigate P3 independently. Do not assume one root cause for all missing parts.
• All parts present in raw generation, only P1 displayed: detect API output length, UI formatting/collapsing or streaming stop. Fix rendering/transport, not retrieval.
• Evidence missing for a genuinely unanswerable part: say so explicitly; a response can be complete about the user's request by acknowledging a missing authoritative answer, but this must not be counted as fully factual coverage.
• Wrong facts joined across parts: bind entity, role, effective date and source version; a part-2 answer from employee policy cannot be grafted onto a contractor question.
• One question asks for an overall comparison: answering each component independently may still fail to synthesize the comparison. Preserve cross-slot dependencies and add a final comparison/decision obligation.

5. Numerical example: stage-by-stage missing-part attribution

• A labelled test set contains 100 three-part questions = 300 answerable subquestion slots, all with verified authorized evidence in the frozen corpus. Assume no abstention/ACL exceptions for this experiment.
• Baseline evidence funnel: planning includes 280/300 slots; retrieval finds evidence for 255/300; reranking preserves 240/300; the exact final prompt contains sufficient evidence for 225/300; final answers correctly and supportably address 180/300. These are nested counts, and first-loss totals are planning 20, retrieval 25, reranking 15, context assembly 15, and generation 45—120 missing slots altogether.
• Thus retrieval-related early loss (planning + retrieval + selection + context assembly) accounts for 75 missing slots; generation-side omission accounts for 45. The difference matters because adding an instruction to “answer all three parts” cannot restore 75 absent evidence slots.
• Illustrative production incident: composite retrieval finds P1 only; running each authorized subquery individually finds P1, P2 and P3. With the original prompt and oracle spans for all three, the model answers P1 and P2 but omits P3—both query dilution and generation incompleteness are present.
• Apply query decomposition, per-slot candidate coverage and protected evidence packing. In a controlled rerun on the same 300 slots, suppose the prompt contains evidence for 285/300 = 95% instead of 225/300 = 75%.
• With a structured answer obligation ledger, correct grounded slot completions rise from 180/300 = 60% to 270/300 = 90%, an absolute +30 percentage points. The remaining 15 absent-evidence slots cannot be answered from the provided prompt; another 15 have evidence but are still missed or misanswered by generation.
• The all-three-parts complete rate is measured separately: suppose 32/100 = 32% at baseline and 80/100 = 80% after repairs. You cannot derive this question-level figure solely from average per-slot recall, because missing facts may cluster within the same questions.
• A planned 3-query fanout increases per-request retrieval calls from 1.0 to 2.4 on average, and p95 end-to-end latency from 210 ms to 340 ms, hypothetical +130 ms. Decomposition and extra validation should be conditional on multipart intent and constrained by budget.
• All figures are illustrative only, not MachinoAI measurements or empirical vendor benchmarks. Validate exact labels, confidence intervals, source freshness, latency and monetary cost in actual canary cohorts.

6. Engineering solution: slot-aware retrieval and synthesis

• Intent decomposition contract: parse each multipart query into a stable list of required intents and dependency edges. Preserve original entity, user scope, negation, fiscal date, units and comparison instructions during decomposition.
• Coverage-oriented planning: assign each slot an authorized source set, query variant, expected type of evidence and budget. When source routing fails, do not silently use a topically similar but non-authoritative source.
• Parallel targeted searches: run independent subqueries when useful, but share snapshot, tenant, ACL and effective-date constraints. Azure's agentic retrieval describes decomposition into targeted parallel queries and returning grounding references/activity for traceability.
• Per-slot evidence inventory: attach slot_id → [source_id,version,section,span] to each supported candidate. One passage may support several slots; keep it once and map all applicable slots to avoid waste.
• Protected packing: reserve token budget for *at least one current authoritative span per answerable slot* before filling the remaining budget with ancillary context. If budget is inadequate, record missing slots and use bounded refetch, hierarchical synthesis or transparent partial answer.
• Structured synthesis: produce an internal object {P1:{answer,evidence_ids,status}, P2:{...}, P3:{...}}; the display layer can render concise paragraphs/bullets. Do not let the model omit a key entirely without returning unsupported/unknown and a reason.
• Deterministic coverage gate: compare output keys against the original slot ledger; ensure each answer is actually supported by its cited spans and that users are told about unanswered components. A valid slot ID is not enough if its text is wrong.
• Grounding verifier: run exact checks for numbers, role, dates and source IDs; use an evaluated entailment checker for paraphrased claims. Link the answer to *authorized model-visible* evidence. An unsupported answer should be blocked rather than filled from model memory.
• Bounded repair: for one missed slot, retry only the targeted authorized retrieval or regenerate from a fixed verified context; do not endlessly rewrite the whole answer or broaden to forbidden sources.
• Observability: log query_id, slot_id, planner_status, retrieval_found, rerank_kept, prompt_included, generation_covered, source_version, citation_support, failure_reason to attribute loss. Avoid storing sensitive raw user text by default.
• Safe canary: compare slot-level and all-slots coverage against a gold set; gate on groundedness, wrong-source citations, false “complete” assertions, p95/p99, token costs and ACL protection. Feature-flag decomposition and the answer contract separately to isolate their effects.

7. Trade-offs and common mistakes

• Decompose every query: raises cost and can damage simple questions. Use intent detection and preserve dependency structure.
• Split into too many subquestions: causes excessive calls, duplicate evidence and extra joins. Deduplicate semantically identical slots, but never merge distinct requested outcomes.
• High top-k without decomposition: may return many near-identical P1 paragraphs while ignoring P2/P3. Evaluate per-part evidence coverage, not only average query relevance.
• Generation checklist alone: can make empty promises for unseen evidence. Mark missing slots as unavailable rather than hallucinating.
• Oracle injection misinterpretation: if gold evidence fixes the answer, generation may still fail with noisier real contexts; benchmark both oracle and realistic retrieval conditions.
• Groundedness vs completeness: high faithfulness for the one stated cap cannot substitute for addressing the approval and receipt parts. Measure both dimensions.
• Longer answers: can include more facts but may increase unsupported claims, verbosity and latency. Optimize correctly supported obligations per response.
• Strict all-slot success: important for procedural tasks but can unfairly penalize safe explicit abstention when a source truly lacks an answer. Report complete factual answers and honest part-level uncertainty separately.
• Multi-source authority: evidence from an unrelated HR policy cannot fill a vendor-contract slot because of text similarity. Enforce source relevance and authorization.
• Asynchronous fanout: parallel queries can return on different index revisions; pin source snapshots and document applicability to avoid stitched inconsistent facts.
• Generation model stochasticity: a single oracle run does not establish deterministic coverage. Repeat frozen-prompt trials and assess material omissions, not text identity.
• UI-level truncation: if output exists but isn't displayed, expanding retrieval or changing prompts cannot solve it.
• Statistics: a 90% slot-success average can hide a much lower all-parts-complete rate. Report both, confidence intervals and high-severity missing obligations.

8. Production monitoring and regression tests

• Maintain expert-labelled multipart questions with two, three, and five components; include mixed source types, prerequisites, comparisons, numeric versus procedural answers, negative constraints, nested clauses and genuine unanswerable parts.
• Add subquestion planning recall, per-slot retrieved evidence Recall@k, rerank and final-prompt survival, per-slot supported answer rate, all-required-slots complete rate, unanswered-slot disclosure, wrong-source citations and cost.
• Inject faults separately: omit P2 from planner; remove P3 from candidate list; clip P2 during packing; hide P3 near middle; force low output tokens; drop the generated P2 block in post-processing. The first-loss telemetry should identify each injected defect.
• Run offline fixed-corpus retrieval-only, oracle-context generation-only and end-to-end evaluations on every material query planner, source routing, reranker, model, prompt, tokenizer or context-budget change.
• Review a sample of automated slot extraction and coverage labels with domain SMEs. LLM judges can miss negation, role, time and modality; evaluate false positives when detecting apparent “completion.”
• Guard security: disallow retrieving an unauthorized source merely to make all slots appear complete, and invalidate cached slot evidence when documents are retired or ACLs change.
• Canary with rollback conditions based on critical missing-part escape rate, wrong or fabricated answers, unexpected source fanout and p95/p99 regression.
• Keep user-facing response concise: show each requested part and, where evidence is missing, explicitly state which part could not be verified.

Interview takeaway: If a multipart RAG answer addresses only the first part, decompose the question into answer obligations and trace each one through planning, retrieval, reranking, prompt assembly, generation and rendering. Compare subquery retrieval with oracle-context and frozen-prompt tests to prove whether a specific part lacked evidence or the model failed to use it. Fix the first failing boundary; ensure every part receives a supported answer or an honest, explicit uncertainty status.

Research references

• Microsoft Azure Architecture Center, RAG information retrieval and decomposition: https://learn.microsoft.com/en-us/azure/architecture/ai-ml/guide/rag/rag-information-retrieval
• Microsoft Azure AI Search, Agentic Retrieval Overview: https://learn.microsoft.com/en-us/azure/search/agentic-retrieval-overview
• Microsoft Azure Architecture Center, RAG End-to-End Evaluation and Completeness: https://learn.microsoft.com/en-us/azure/architecture/ai-ml/guide/rag/rag-llm-evaluation-phase
• Microsoft Foundry, RAG Evaluation—Groundedness vs Response Completeness: https://learn.microsoft.com/en-in/azure/ai-foundry/concepts/evaluation-evaluators/rag-evaluators
• Amazon Bedrock, Knowledge Base Evaluation Metrics: https://docs.aws.amazon.com/bedrock/latest/userguide/knowledge-base-evaluation-metrics.html