A retrieved clause states an exception, but the generated response mentions only the general rule. How would you prevent missing qualifications?

Production incident (illustrative): An employee asks, “Can a Singapore contractor claim a hotel above the SGD 350 nightly cap for a conference?” The assistant retrieves the current policy and confidently answers: “Contractors may claim accommodation only up to SGD 350 per night.” It cites the right 2026 policy. But the same authorized, model-visible clause says: “For a pre-approved accessibility requirement, the cap may be exceeded with written Regional Finance Director authorization before booking.” The user has stated an accessibility-related need. The assistant’s general rule is literally supported yet its response omits an applicable exception, potentially causing an incorrect denial.

Key distinction: Faithfulness is not sufficient for completeness. The included sentence may be true, but the answer as a whole can misrepresent the policy by omitting a condition, exception, exclusion, negation, override, effective-date restriction or approval path. Microsoft’s RAG evaluation guidance distinguishes groundedness (claims do not exceed the context) from response completeness (critical expected information is not missing). Evaluate whether a *qualified decision rule* was represented, not just whether each stated claim sounds supported.

1. Contain the incident and verify what the model actually saw

• Check the user question, entity/role, jurisdiction, date, requested action, source versions and authorization. Is the exception actually applicable, or is the user asking only for the default cap? Do not append every remote exception to every response.
• Trace the full evidence funnel: authoritative document → indexed chunk → retrieved evidence → reranker → compressor → final serialized prompt → answer and citations. Record the exact paragraph text with source/version/offset and each output stage’s hash.
• If the exception was dropped during compression or context packing, it is a Q266 context-survival defect. If the exception survives in the final prompt but the model ignores it, this is Q269 answer-qualification omission. If the model makes an unsupported claim, refer to Q268’s claim-level citation checks.
• Establish authoritative precedence: which rule and exception are current; whether the exception overrides the limit or merely provides a process to request review; whether approval is prior to booking. A wrong exception can be more harmful than no exception.
• In consequential policy guidance, disable unconditional “never allowed” conclusions until applicability is checked. Return an uncertainty statement or escalate if controlling clauses cannot be reconciled.

2. Extract a structured rule-plus-exception contract

• Represent a rule as rule_id, source_id, version, scope, condition, default_effect, exception_id, exception_condition, exception_effect, required_action, approving_authority, effective_period and precedence.
• Example default: role=contractor AND location=Singapore → hotel cap SGD350/night. Exception: preapproved_accessibility_need=true AND written_Regional_Finance_Director_approval_before_booking=true → may_exceed_cap. This is permission conditional on approval, not automatic entitlement.
• Make the condition scopes explicit. “Except in emergencies” may modify a time deadline rather than an amount cap; grammar and section layout matter. Preserve conditional verbs such as must, may, unless, only if, except, subject to, notwithstanding and provided that.
• Extract source sentence offsets for default and exception, plus neighboring definitions, headings, footnotes and referenced sections. When the exception refers to “the above threshold,” include enough nearby text to resolve the reference.
• Avoid blanket exceptions based on similar words from obsolete policy versions, another tenant, employee rather than contractor provisions, or an entirely different region.
• Treat structured extraction as a hypothesis: validate important legal/finance clauses with a deterministic field matcher or a calibrated independent reviewer, and record ambiguous cases for human policy ownership.

3. Root-cause debugging sequence

• A. Evidence eligibility: confirm the exception clause is in an authorized, active, governing source and relates to the requested user. A legitimate exclusion by jurisdiction or date is not an omission bug.
• B. Context preservation: verify default and exception spans survive search, deduplication, rerank, compression and packing. A high-scoring general-policy summary may crowd out the only exception sentence.
• C. Source segmentation: check whether chunk boundaries separate “General rule” from “Exceptions,” whether a table footnote was dropped or a section heading misclassified. Index section/parent linkage and retrieve exception neighbors when needed.
• D. Intent binding: compare user attributes and explicit qualifiers to rule preconditions. The planner may identify a contractor hotel-cap question but not the stated accessibility requirement.
• E. Retrieval/selection bias: inspect whether a short generic FAQ outranks the detailed exception. The answer model may prefer prominent, repetitive general rules; check source authority and facet-level coverage.
• F. Generated answer audit: extract the answer’s unconditional claims and compare to rule schema. “The cap is always SGD350” conflicts with a possible approved exception; “Normally SGD350, with a documented accessibility exception subject to prior written approval” preserves the decision structure.
• G. Complementary omission evaluation: distinguish an answer that is supported but incomplete from one that contradicts an exception by saying “never.” Standard claim-entailment precision tests may pass the former; add a required-fact recall and condition-coverage check.
• H. Repair loop: if the default is present but an applicable qualifier is omitted, regenerate with the validated default+exception spans and explicit conditional structure. Recheck both groundedness and completeness, then enforce a bounded retry limit.
• I. Later-stage issue: if the exception is represented but combined with another person's facts or an incompatible reporting period, investigate entity/temporal consistency (Q270–Q271) rather than simply changing sentence order.

4. Engineer a qualification-aware answer pipeline

• Rule graph/ledger: for higher-risk policy sources, extract or author rule groups linking default → exception → prerequisite → approval → effective version. Preserve original text and provenance; use source owners to approve changes when necessary.
• Query-to-rule matching: derive trusted user constraints (role, region, effective date, purpose) from the question and permitted profile metadata. Classify as default_only, exception_potentially_applicable, exception_confirmed or insufficient_attributes; avoid inferring sensitive conditions not stated.
• Evidence assembly: fetch default and candidate exception clauses together, including necessary definitions and relevant footnotes, while retaining source offsets. Use a bounded neighbor expansion instead of adding an entire unrelated policy.
• Structured generation: ask the answerer to complete default rule, applicability, exception, approval requirement, uncertainty, citations as internal checked fields before producing concise prose. The user-facing response can stay short, but the decision must not silently discard controlling qualifiers.
• Dual validation: (1) groundedness/claim support—every expressed fact is backed by exact model-visible cited spans; (2) *completeness/qualifier recall*—each applicable material condition in the evidence ledger is represented in the answer or explicitly marked unknown.
• Logic guards: deterministic checks for critical numbers, inequality signs, dates, approvers, policy versions, and strong absolute words (“always,” “never,” “automatically”). Mark a violation when an unconditional conclusion is generated while a governing exception exists.
• Abstention and clarification: if the exception requires a fact not supplied (e.g. pre-approval already granted), say that the outcome depends on that fact; do not assume approval, deny it categorically or invent eligibility.
• Authority conflicts: where a global rule and local exception differ, apply documented precedence and effective dates rather than LLM preference for one text. If authority is unclear, flag human review.
• Safety controls: all source/neighbor fetches obey tenant, ACL, redaction, version and time restrictions. Do not show exceptions from protected documents solely because they exist in the same corpus.

5. Numerical example: why a faithful answer may still be wrong

• Build 200 expert-labelled policy questions with verified active evidence. For 120 cases a general rule suffices; for 80 cases an exception or qualification is material to the question. Labels include exact default, applicable condition, override, required approval and citation.
• At baseline, suppose the response states a supported general rule in 188/200 = 94% cases. A naive “is the answer supported by the cited paragraph?” check reports high faithfulness, but only 28 of 80 exception-dependent answers include the controlling qualification: qualification recall = 28/80 = 35%.
• Complete and legally applicable answers total 120 default-only + 28 exception-covered = 148/200 = 74%. The remaining 52 exception cases may be superficially grounded but still incomplete. This arithmetic assumes all 120 default-only answers are correct in the controlled example.
• Add source-linked exception extraction, query applicability matching and a final required-qualifier coverage gate. On the same 80 cases, suppose 68 include the correct conditional exception and prior-approval requirement: 68/80 = 85% qualifier recall.
• If the 120 default-only answers remain correct, complete applicable answers become 120 + 68 = 188/200 = 94%, a 20-percentage-point gain in complete-answer rate versus 74%. Ten additional errors remain for specific root-cause review.
• The extra gating pass adds a hypothetical p95 165 ms → 260 ms (+95 ms), and 14 of 200 initial answers may request clarification or be conservatively abstained where approval facts are unknown. Those abstentions must be reflected separately in answer-utility evaluation rather than counted automatically as successes.
• All figures are illustrative, not measured MachinoAI operational metrics or claims from the cited research. Human adjudication and a carefully controlled holdout are needed before setting a deployment threshold.

6. Evaluation metrics and test design

• Applicable qualification recall: number of required, governing qualifiers represented correctly / number of required governing qualifiers across labelled questions. Evaluate *applicability*, not just mention counts.
• Complete-answer rate: fraction of questions whose answers include the default and all material conditions and are valid for user scope. Record wrong-effective-date and wrong-policy outcomes separately.
• False-exception inclusion: percentage of answers incorrectly adding an exception that does not apply. Optimizing only exception recall can produce irrelevant or dangerous overqualification.
• Unconditional-overclaim rate: outputs using “always/never/automatically” when the governing policy includes an applicable exception.
• Evidence coverage by stage: default+exception available at retrieval, present after context packing, identified in the rule ledger, then reflected in the final answer.
• Groundedness and citation support: verify included conditions are cited to the governing text; a complete-sounding answer can still hallucinate policy details.
• Decision accuracy and clarification: evaluate whether the resulting action is appropriate, and whether the assistant asks for missing facts rather than assuming them.
• Use paired tests for policy rules with/without an exception, cases where a qualifier changes the outcome, negations, overlapping jurisdictions, dates, permission versus automatic entitlement, and user questions intentionally asking only for a simple baseline.
• Run separate source-verification and output-verification gold sets. The same model extracting, generating and grading may share mistakes; independently adjudicate a risk-weighted sample.

7. Engineering trade-offs and limitations

• Show every exception: technically easy but bloats answers, adds irrelevant caveats and may confuse users. Include only material or plausibly applicable qualifiers, with optional deeper details.
• Short answers: better UX but more likely to omit caveats. A concise two-sentence “normally X; if Y, then Z with approval” can be both short and complete.
• Full-document context: could contain all exceptions but increases noise and token costs. A bounded linked-rule extraction is more targeted, provided offsets and scope remain correct.
• LLM rule extraction: flexible for prose and footnotes but imperfect for nested conditions, tables and modality. Validate high-risk policies with rule ownership and exact text.
• Rigid templates: enforce coverage but may sound repetitive and fail unusual document structures; render natural language after the structured facts are verified.
• Prior approval: “may exceed with approval” does not mean “automatically reimbursable”; guard against modality inversion.
• Omitted versus contradicted: an incomplete general-rule answer and an unconditional “never allowed” answer have different severity. Metrics and remediation should reflect that.
• Multiple exceptions: an exception can have sub-exceptions or override conditions. Preserve a precedence chain and avoid flat keyword lists.
• Policy drift: newly added exceptions can change previously correct answers even if retrieval of default rules is stable; version and re-evaluate the rule ledger.
• Evaluation bias: synthetic questions may make exceptions more obvious than real user queries. Include actual anonymized phrasing, reviewer-validated hard negatives and ambiguous cases.
• Latency/cost: extraction, rule checks, and regeneration add work; tier by consequential decisions and measure p95/p99 plus abstention burden.

8. Prevention and production runbook

• Every indexed policy update gets a default/exception/condition audit. Changes in caps, prerequisites, approvals, negation or effective dates invalidate old rule schemas and cached answers.
• Maintain a clause-aware regression suite: “usually” vs “always”; “unless” vs “except”; “may request approval” vs “approved”; boundary amounts, pre-/post-booking dates, missing permissions, wrong regions and retired versions.
• Emit secure trace fields: source_version, rule_id, exception_id, default_span_id, qualifier_span_id, qualification_applicable, qualifier_in_prompt, qualifier_in_answer, judge_verdict, citation_ids, and any clarification.
• Alert on a growing retrieved-qualifier → omitted-answer gap, increased unconditional-overclaim rate, critical wrong-policy decisions or new extraction errors after ingestion/prompt/model changes.
• Test at multiple levels: source extraction; retrieval of linked clauses; context-budget survival; conditional inference; final answer completeness and citation verification.
• Canary rule-aware generation on real production-like query slices and evaluate complete-answer accuracy, qualifier recall, false exceptions, abstention, latency and cost before rollout. Preserve rollback and current ACL enforcement.
• Require SME adjudication for critical policy families and disagreements between a general rule and purported local exception. When source precedence is unknowable, show uncertainty rather than silently choosing one.

Interview takeaway: A RAG answer may be entirely faithful to the words it repeats yet still misleading because it omitted a controlling exception. Build source-linked rule/exception extraction, test whether the exception applies to the specific query, and gate outputs on both claim support and required qualification recall. If the clause is in the prompt but absent from the answer, fix conditional reasoning and completeness—not vector search.

Research references

• Microsoft Foundry, RAG evaluators (groundedness vs response completeness): https://learn.microsoft.com/en-us/azure/foundry/concepts/evaluation-evaluators/rag-evaluators
• Microsoft Azure Architecture Center, RAG prompt engineering and self-evaluation: https://learn.microsoft.com/en-us/azure/architecture/ai-ml/guide/rag/rag-prompt-engineering
• Microsoft Azure Architecture Center, RAG LLM evaluation phase and completeness: https://learn.microsoft.com/en-us/azure/architecture/ai-ml/guide/rag/rag-llm-evaluation-phase
• Es et al., RAGAS: Automated Evaluation of Retrieval Augmented Generation (EACL 2024): https://aclanthology.org/2024.eacl-demo.16/