Users report that answers are accurate but consistently omit important details. How would you measure and debug completeness?

Production incident (illustrative): A procurement RAG assistant answers, “What must our team do before booking a Singapore contractor's hotel above the SGD 350 limit?” It correctly says “obtain written Regional Finance Director approval”, and cites the governing policy. But it routinely omits three material details: an accessibility or documented business justification, *approval before booking*, and storing the approval with the reimbursement claim. Users report that the answers are accurate but not actionable. The underlying clauses exist in the authorized corpus. The incident is systematic answer incompleteness, not necessarily a hallucination.

Core distinction: Groundedness/faithfulness asks whether the information *stated* is supported by source evidence (precision-like). Completeness asks whether the response covers all material, applicable requirements of the question and trusted ground truth (recall-like). A short, true answer can fail the user's task. Microsoft Foundry distinguishes groundedness from response completeness, and Amazon Bedrock evaluates how well an answer resolves *all aspects* of a question. Measure both, plus verbosity and citation validity.

1. Contain the incident and define what a complete answer means

• Establish exact user intent, permissions, domain and risk. “What is the cap?” may need a single number; “How do I request an exception?” needs eligibility, approval authority, timing, documentation and next steps. Do not demand unrelated facts simply to maximize coverage.
• Acquire an authoritative, current, permission-appropriate answer specification from source owners, including mandatory and optional facts. For each expected fact, record a minimal supporting span and applicability conditions. Separate *material omissions* from stylistic brevity.
• Trace the specific request through source snapshot, query planner, source selection, retrieved candidates, reranker, dedup/compression, final model-visible context, structured reasoning and final response/citations. Retain source IDs, versions, user role, offsets and a secure prompt manifest.
• Hold unsupported or incomplete high-impact instructions for bounded refetch, targeted clarification or escalation. A partial finance-policy answer should not be presented as the full procedure when omitted prerequisites could change the decision.
• Diagnose overlapping prior lessons precisely: Q266 covers retrieved sentence lost during trimming, Q267 covers position sensitivity despite prompt presence, Q268 covers unsupported stated claims, Q269 covers a specifically omitted conditional exception, and Q275 will isolate coverage for multipart questions. Q273 establishes general answer completeness measurement and stage-by-stage root cause across facts, workflows and question types.

2. Build an answer-fact coverage contract

• Represent expected content as structured atomic answer units: requirement_id, question_intent, fact_or_step, required_if, importance_weight, source_id, version, source_span, authority, answer_support_criteria.
• Example booking-exception checklist: F1: default SGD350/night cap; F2: qualifying accessibility/business case; F3: Regional Finance Director written approval; F4: approval obtained BEFORE booking; F5: retain approval with expense claim.
• Assign weights reflecting consequences, not word count: F1=1, F2=2, F3=3, F4=3, F5=1. A missing required approver or deadline should count more than a missing introductory definition.
• Define applicability: a case requiring approval has F2–F5; a user asking only for the ordinary cap may not. Avoid penalizing answers for excluding facts outside the user's scope.
• Include dependency relationships and acceptable paraphrases. “Written sign-off before booking” may cover approval medium/timing when tied to the correct authority, but “finance must review” does not prove the Regional Finance Director requirement.
• Record authority/time/region/role. A fact about an obsolete employee policy is not “covered” for a 2026 contractor question, even if it looks similar.
• Allow “cannot determine from authorized evidence” as a calibrated uncertainty outcome, never as equivalent to a fully covered answer. Track answerability and abstention separately.
• Human adjudicators should approve gold fact sets; LLM-extracted coverage units are useful candidates, not authoritative truth. Keep disagreements and changes versioned.

3. Evaluate completeness at fact, answer and task levels

• Atomic fact coverage: count expected applicable fact units correctly expressed and supported; fact_recall = covered_required_facts / total_applicable_required_facts.
• Weighted material completeness: sum(weights of correctly expressed mandatory facts) / sum(weights of applicable mandatory facts). In the example, the assistant mentions F1 and F3 only: (1+3)/(1+2+3+3+1)=4/10=40%; mere sentence-level accuracy can still be high.
• All-critical-facts-present rate: a binary per-question gate requiring every high-severity unit, regardless of weighted average. An answer missing “before booking” may be an unsafe instruction even if its remaining coverage is 90%.
• End-to-end task resolution rate: domain reviewers confirm that the response gives the required procedure or decision without invented steps. Distinguish completeness from correctness and user satisfaction.
• Unsupported-claim rate / citation precision: measure alongside fact recall; adding generic or fabricated content can raise length and apparent coverage without improving grounded completeness.
• Coverage precision / verbosity: label unnecessary, irrelevant or repeated facts; an answer can be complete yet unwieldy. Report useful facts per 100 output tokens and response latency.
• Per-stage evidence coverage: what fraction of gold fact units is in the source, retrieved candidates, reranked set, final prompt and generated answer? This localizes the first lost fact.
• Missing-fact severity and slice metrics: report by domain, source/section, question length, user intent, policy exceptions, tables, language, factual/procedural/comparative questions, and token budget. A high overall metric can hide repeated omissions in one critical workflow.
• Evaluator reliability: use normalized propositions with subject/time/condition bindings and source-linked passages; human-score a calibration set to compute detector precision/recall and false positives. One LLM judge is not a perfect arbiter of “important.”

4. Root-cause debugging: localize the missing information

• A. Gold/source gap: if a required fact is missing or outdated in the knowledge base, route to source owners; retrieval cannot produce evidence absent from the authorized corpus.
• B. Planning gap: did the system parse “what do I need to do” as one fact (“who approves”) instead of a full process? Compare the intended sub-goal checklist against planned queries. For general completeness, ensure workflow steps are represented before retrieval.
• C. Retrieval gap: if gold spans exist in the index but are absent from candidates, compare per-fact Recall@k, dense/BM25 branches, filters, query rewriting and parent-section expansion. Increasing top-k alone may add redundant generic passages without the missing footnote.
• D. Rerank/selection gap: if F4 was retrieved but pruned during reranking or source deduplication, inspect score versus novelty, group-by source, candidate budgets and rank-window decisions.
• E. Packing/compression gap: if the final prompt lacks the approval timing or document retention sentence, compare chunk hashes, source offsets, token budgets and dropped-span reason codes (Q266). Preserve relevant adjacent sentences and headings.
• F. Reading/attention gap: if all required facts appear in the prompt but those in the middle are overlooked, test controlled position swaps and context reduction (Q267).
• G. Generation gap: if the prompt clearly contains all facts and the answer omits F2, F4 and F5, inspect “answer briefly” instructions, response token caps, template constraints, under-specified completion criteria and miscalibrated summarization. Compare a structured fact-ledger generation with the current free-form response.
• H. Post-processing gap: a streaming cutoff, markdown formatter, answer summarizer, UI character cap or citation renderer may remove sections after generation. Compare raw model output to displayed response.
• I. Evaluation gap: a metric measuring only answer relevance or claim groundedness may label the short response good. Inspect evaluator prompts, reference labels, weighting and human disagreement.
• J. Ambiguity gap: if the question omits facts necessary to determine requirements, the system should surface the missing attribute or ask one targeted question—not guess or claim full completeness.

5. Beginner-friendly numerical production example

• Create 200 authorized, expert-labelled enterprise questions. Each has exactly 5 applicable required answer units, for a total of 1,000 mandatory fact opportunities. This is a controlled illustration, not MachinoAI production data.
• Suppose the current system includes and supports 700/1,000 = 70% required units, even though 190/200 = 95% of its *stated* claims are judged grounded and mostly accurate. Distinct denominators explain why accuracy/faithfulness scores can mask omissions.
• Stage-wise, the source corpus has 1,000/1,000 units; retrieval finds 940/1,000 = 94%; reranking retains 900/1,000 = 90%; final context contains 820/1,000 = 82%; the generator expresses only 700/1,000 = 70%. Lost units are: retrieval 60, rerank 40, packing 80, generation 120. These stage losses are disjoint by *first loss* in the illustrative funnel and total 300.
• The team repairs the dominant first-loss stages: coverage-aware retrieval/selection, span-preserving context packing, and structured required-fact checking. Suppose retrieval reaches 970, rerank 950, context 930, output 900 required facts out of 1,000, for 90% fact recall—a 20-percentage-point gain.
• On the same 200-question set, suppose all-required-facts answer success improves from 68/200 = 34% to 142/200 = 71%. This is not derivable from aggregate fact recall; it is measured separately on question-level labels.
• For the five-unit hotel example, the original answer contains F1+F3 (weighted 4/10 = 40%); the repaired answer contains all five (weighted 10/10 = 100%) while preserving accurate citations and conditional applicability.
• Added evidence checks increase illustrative p95 latency from 185 ms to 275 ms, a 90 ms overhead, and output tokens from 110 to 155 on average. Track costs and critical-fact omission rate, not only raw coverage; a one-line correct answer may remain preferable for a one-fact question.
• Do not game the metric by producing longer lists: a generated “fact” counts only when applicable, correct, supported and substantively responsive to the requested task.

6. Engineering solution: fact-coverage-aware answer assembly

• Plan expected outputs: classify question type and derive a small scoped answer checklist (steps, prerequisites, constraints, actors, quantitative facts, timelines or comparisons). Ground the checklist in authoritative source labels rather than generating arbitrary “nice-to-have” details.
• Retrieve for coverage: use subqueries or controlled section expansion for distinct checklist items; compare required-fact coverage of candidate sets, not only top-k relevance. A fact-aware selector can prefer one relevant exception paragraph to five duplicate summaries.
• Preserve provenance: attach stable fact_id, document_version, source_span, question_requirement_id to every candidate; carry them through rerank, compression, packing, answer and citations.
• Budget by important facts: allocate context for high-priority, nonredundant evidence; avoid trimming the only sentence supporting an important prerequisite. Use the real model tokenizer and verify the final serialized request.
• Generate from a coverage ledger: construct a structured internal mapping required_fact → supporting_span → covered/unknown before writing natural language. Then render a concise user response with clear steps and citations; do not expose irrelevant ledger detail.
• Pre-release completeness gate: compare generated answer against the validated required-fact ledger using deterministic checks for amounts, names, dates, enums and an entailment/LLM judge for paraphrases. Block critical omissions, allow controlled refetch/rewrite, and reverify.
• Separate omission from hallucination: a verifier should grade both precision of stated claims and recall of required facts, plus question-level all-critical completion. A correct but incomplete answer should trigger a targeted addition, not an indiscriminate rewrite.
• Risk-aware escalation: if a critical fact remains unavailable after bounded retries, state the limitation or ask a clarification rather than improvising. Enforce permissions during all source expansion and repair loops.
• Feature-flag rollout: evaluate against a held-out gold set and shadow traffic, then canary by domain with audit samples for fact recall, all-critical coverage, wrong/unsupported facts, output verbosity, p95/p99 and user task success.

7. Trade-offs, failure scenarios and common mistakes

• Longer answers can contain more facts but also add noise, unsupported assertions and tokens; optimize relevant supported facts per answer, not length.
• Generic completeness prompts (“include all details”) rarely define what is material. Use task-specific, authoritative expected-fact specifications.
• Reference-free grading is flexible for unlabelled traffic but cannot reliably distinguish missing facts from facts that were never applicable. Calibrate against human-labelled gold answers.
• Aggressive subquery expansion increases recall but also latency, cost and opportunities for contradictory policies; require a bounded, authorized source set.
• One chunk per document can save tokens but delete separate required sections. Prefer span-aware selection and document-linked neighboring paragraphs.
• Fact weights reflect business risk but can be subjective; owners should approve them, and reports should show weighted and unweighted recall rather than hiding severe omissions in one aggregate.
• Coverage judge errors include false claims of support due to overlapping keywords, paraphrase rejection and missed negation; test numeric and modality perturbations.
• Question ambiguity may require clarification; do not label a justified uncertainty as an omission when the source genuinely cannot determine the answer.
• Overfitting to reference text can penalize valid shorter paraphrases; compare semantic claim coverage with stable entity, time and conditional scope.
• Source authority conflicts can yield incompatible facts; the correct response may need to state the disagreement instead of “covering” both as simultaneously valid.
• UI truncation may hide output sections even if the generator produced them; check the rendered user experience.
• Stable partial answers may look consistent in repeated runs (Q272) yet remain incomplete. Consistency and completeness are independent dimensions.

8. Monitoring and prevention

• Maintain a gold requirement checklist for representative workflows and critical policy decisions, with source/version offsets, expected units, applicability and reviewer-approved weights.
• Capture stage-level required_fact_id survival: source present → retrievable → reranked → packed → generated → displayed. Log first-loss reason categories and compare with independently audited samples.
• Dashboard unweighted and weighted fact recall, all-critical-facts present rate, end-to-end task resolution, unsupported-claim rate, citation precision, verbosity, abstention and p95/p99 by domain and question type.
• Build regression cases for missing prerequisites, dates, units, approval timelines, multi-step processes, cross-document instructions, tables/footnotes, negation and hard token caps.
• Inject controlled failures: remove a required paragraph from retrieval, clip a context chunk, insert repeated summaries, set low output max tokens, and truncate the UI; assert that alarms and safe fallback trigger.
• Re-run coverage checks after policy changes, source owner updates, indexing/chunking, reranking, context compression, prompt-template changes and model upgrades. Previously complete answers can become outdated.
• Human review high-impact, highly confident but incomplete responses to catch issues automated judges miss; add these failures to the benchmark.
• Guard against data leakage by evaluating only evidence permitted to the requester; missing *unauthorized* details must not be counted as completeness failures.
• Define deployment gates on critical-fact omission escapes, grounded accuracy and answer usefulness, not a single aggregate judge score.

Interview takeaway: “Accurate but missing important details” is a recall/completeness failure, not necessarily a hallucination. Define the applicable required-fact set, measure its survival from corpus through retrieval, reranking, packing, generation and display, and repair the first stage that loses each fact. Use both groundedness (stated-fact precision) and completeness (required-fact recall), with critical-fact gates and verified citations.

Research references

• Microsoft Foundry, *RAG Evaluators—Groundedness and Response Completeness*: https://learn.microsoft.com/en-us/azure/ai-foundry/concepts/evaluation-evaluators/rag-evaluators
• Microsoft Azure Architecture Center, *RAG LLM End-to-End Evaluation Phase*: https://learn.microsoft.com/en-us/azure/architecture/ai-ml/guide/rag/rag-llm-evaluation-phase
• Amazon Bedrock, *Use metrics to understand RAG system performance*: https://docs.aws.amazon.com/bedrock/latest/userguide/knowledge-base-evaluation-metrics.html
• Amazon Bedrock, *Retrieving information from data sources*: https://docs.aws.amazon.com/bedrock/latest/userguide/kb-how-retrieval.html