A model starts producing invented source URLs despite receiving valid document IDs. How would you make citation generation deterministic and verifiable?

Production incident (illustrative): An internal procurement RAG assistant correctly retrieves doc_id=policy-sg-2026-v3 and a source excerpt stating that Singapore contractor travel exceptions require written Regional Finance Director authorization. It generates a useful answer but cites https://policies.example.com/sg/contractor-expense-2026—a URL that was never returned by retrieval or stored in the document registry. In production, a similarly plausible path happens to resolve to an unrelated outdated page. Users trust the clickable link, despite the citation being fabricated. The correct document identity existed throughout retrieval; the defect occurred when the model was permitted to *invent the presentation URL*.

Core design rule: A model can decide which allowlisted source reference IDs support a claim, but it must never be the authority that invents or resolves a citation URL. Build a trusted, deterministic reference registry → validated citation ID → authorized document/span resolver → link renderer. Distinguish *citation existence*, *source authorization*, *URL integrity*, *source version*, *actual claim support*, and *link reachability*. A link that resolves successfully can still be the wrong citation.

1. Contain the incident and establish the ground truth

• Stop rendering free-form model-generated links as verified citations. If a citation token cannot be resolved to a source in the exact retrieval response, suppress the link, block high-risk answers or give a transparent “source link unavailable” state.
• Collect the source retrieval manifest for the affected request: request_id, tenant/principal, source/document IDs, chunk/section offsets, source version, title, canonical source URI or approved lookup handle, content hash, connector ID, permission scope, and source effective date.
• Compare three artifacts: retriever references, the exact serialized model-visible source IDs, and the final answer's citation IDs/URLs. Determine the first stage where an unsupported URL appears. An LLM-generated citation URL can be fabricated even when document metadata and answer text are correct.
• Revalidate source policy: the user must have access to the underlying record and the intended version; a link must not allow cross-tenant document access or reveal protected source identifiers.
• Never “repair” a fabricated citation by searching the open web for the closest-looking URL: doing so can map an unsupported claim to the wrong document or expose private titles in search queries.
• Record whether the source has a stable external canonical URL, a connector-specific document handle, an ephemeral signed URL, or no user-accessible link at all. Not every document ID should be converted to a public URL.

2. Root-cause debugging: locate the citation failure

• A. Source ingestion: inspect whether a source URI exists, is immutable/canonical or is an unstable file path, and whether the document moved or was versioned. A document ID can remain stable when its original storage location changes.
• B. Retrieval result schema: check whether the retriever returns doc_id, chunk_id, source_id, title and source_uri separately. A display title or relative storage path is not a safe source URL.
• C. Reference mapping: confirm that result-local citation numbers such as [1] map unambiguously to the original returned reference_id, rather than list positions after reranking, deduplication or context trimming.
• D. Context assembly: ensure every included evidence span receives a stable request-scoped citation token mapped to its document/version/span. If a chunk is excluded from the final prompt, its token must not remain available for an unsupported generated citation.
• E. Generation instructions: examine whether a prompt asks the LLM to “include full source URLs.” That asks the model to synthesize information it cannot verify. Replace with an instruction to emit citation reference IDs only from the supplied enumeration.
• F. Decoding/schema validation: ensure free-form markdown links or model-produced url fields are rejected. A JSON schema or tool-output contract can constrain *format*, but a malicious or mistaken permitted-looking URL still requires trusted registry lookup.
• G. Post-processing: inspect regex replacements, URL auto-linkers, markdown renderers and citation index shifting. A formatter can turn a harmless source title into a fabricated URL or associate the right numbered marker with the wrong record.
• H. Redirect behavior: check if a legitimate canonical link now redirects to another tenant, domain, latest-version record, login screen or outdated content. Validate each hop and final provenance; HTTP 200 alone does not prove correctness.
• I. Cached references: stale cached answers or citation maps can survive document deletion, role changes and file moves. Cache by source generation, tenant and access context; invalidate on ACL, version and URI changes.
• J. Answer grounding: a link to the *right file* may still attach to a paragraph that does not support the claim. Source resolution is distinct from Q268's claim-entailment verification.

3. Trusted citation architecture

• Ingestion registry: maintain an authoritative source_id, stable document_id, document_version, chunk_id, optional page and span_offsets, canonical connector handle, approved source URL when available, content hash, source owner, tenant and ACL.
• Request-local evidence set: after retrieval and packing, create ref_A, ref_B etc. Each points to a verified *model-visible* span and immutable evidence metadata. The LLM sees only this allowlisted reference set.
• Constrained output: return structured claims with text and citation_ref_ids: ["ref_A"]. Accept only references in the current request's allowed set; avoid asking for url, hostname, or path in the model output.
• Server-side resolution: a trusted service looks up ref_A → tenant/source/document/version/span, checks authorization and freshness, then obtains an approved URL or an authenticated document-viewer route. The model is not asked to create paths.
• Rendering: the UI converts validated IDs into clickable citations using *only* the resolver's returned destination. If no safe link exists, render a citation title/source label or controlled source preview without a fabricated href.
• Citation granularity: attach citations to the smallest factual assertion where practical, with page/section/offset when supported. Store exact source text hashes to prevent shifts after document changes.
• Document moves and versioning: stable document identity must survive path changes; the registry maps an identity to the current approved location while preserving citations to applicable historical versions when legally allowed.
• Deterministic mapping: sort or assign reference IDs consistently in the frozen evidence manifest. Do not rely on the LLM's token order for URL selection; identical manifests plus authorization state should render the same links.
• Permissions: issue short-lived signed document links only after each request's authorization; never embed bearer tokens or long-lived signed URLs in model prompts or caches. A signed URL may expire without invalidating the underlying citation identity.

4. Worked numerical example

• In an illustrative benchmark, 200 RAG answers contain 600 citation mentions across 120 unique documents. The generation model receives all 120 correct document IDs, but free-form URL generation invents or corrupts 54 of 600 links: 9% invented-link rate.
• Of 54 invalid links, assume 30 return 404, 14 resolve to a different document, and 10 resemble valid links but point to outdated versions. Checking HTTP status alone catches at most 30, missing the 24 semantically incorrect links that resolve.
• The application introduces 600 request-local citation references, accepts only registered IDs, and resolves them server-side. Suppose 588/600 references successfully resolve to the correct authorized document/version, while 12/600 are rejected because the model emits unrecognized citation IDs or the user lacks current access.
• Verified correct-citation delivery rate = 588/600 = 98%. Fabricated URLs rendered = 0/600 = 0% by construction: unresolved citations are blocked, *not counted as verified*. Do not report 100% citation coverage; it remains 98% because 12 cannot be displayed as verified links.
• If the benchmark includes 500 grounded claims, a separate span-support judge finds only 460/500 = 92% supported by cited passages. Thus the 98% safe link resolution does not imply 98% claim faithfulness; this illustrates the distinction between link provenance and claim entailment.
• Example registry: ref_A → (tenant=T1, doc=policy-sg-2026, version=v3, section=4.6). A generator outputs citation_ref_ids=["ref_A"], and the server returns a verified viewer route or permitted original URL. If it outputs ref_Z or a fabricated https://.../2026, the renderer rejects it.
• Suppose p95 increases from 155 ms to 173 ms with registry/permission checks, an illustrative +18 ms. Cache safe reference metadata cautiously while performing current-authorization checks. These counts and latency values are *hypothetical*, not MachinoAI production measurements.
• For a user who has lost access since indexing, ref_A must fail closed even if the source still exists; availability is not authorization.

5. Engineering implementation

• Step A — preserve provenance at ingestion: assign immutable source/document IDs independent of file path or title; store version, content digest, connector-native handle and verified external URL only when supplied by a trusted connector or repository.
• Step B — produce a retrieval manifest: after filtering and context packing, record precisely which authorized chunk spans were sent to the model. Include a request-local allowlist of citation IDs and hashes; treat out-of-context citations as invalid.
• Step C — ask for structured references: request an answer with statement-level citation_ref_ids drawn from the allowlist. Constrained schema output is helpful, but membership is validated separately on the server.
• Step D — validate IDs and support: reject unknown references, duplicate malformed IDs, references to dropped chunks, wrong-version source IDs and unauthorized documents. Optionally verify the cited span supports the statement (Q268).
• Step E — resolve with a trusted registry: map IDs to canonical URLs or a secure document preview route. Use server-generated URLs, not model output, and never accept hostnames/paths from the LLM as destinations.
• Step F — enforce safe link policy: only approved schemes (https for external destinations), expected domains/connector routing, no javascript:/data: or localhost/internal endpoints, and validated redirect behavior. Avoid URL fetches to arbitrary destinations to prevent SSRF. An allowlisted domain is a defense but not a substitute for source identity.
• Step G — handle absent URLs: return source title, section and a verified in-product preview or a non-clickable provenance label. Do not guess URL patterns from a document name.
• Step H — repair in bounded steps: if a generated citation ID fails, attempt targeted regeneration with the same frozen valid reference set or remove the unverifiable citation and mark the answer limited. Cap retries to avoid loops; never silently substitute a nearby document.
• Step I — secure cache and auditing: store answer_id, manifest_hash, citation_ref_id, source version, resolution outcome, authorization decision, original connector link source, final rendered URL and timestamp; exclude tokens/secrets and respect retention.
• Step J — deploy in phases: shadow-resolve generated citations first, compare existing and resolved destinations, then canary deterministic rendering; monitor malformed IDs, rejection, broken-link rate, wrong-document rate, provenance mismatches and latency with a rollback that never reinstates unsafe free-text URLs.

6. Failure scenarios and trade-offs

• Valid-looking but wrong link: URL syntax or HTTP 200 does not prove the linked document supports the claim. Verify document/version identity using registry metadata and optional source hash.
• Valid ID with no user-facing URL: the source can be cited via controlled internal preview or title/section. “No URL available” should not be hallucinated away.
• Mutable file path: renames and folder moves break path-derived URLs; use stable document IDs and a resolver (as in Q239).
• Wrong document behind redirect: a trusted source link may later point to another record; validate endpoint/identity and redirect policy, not only reachability.
• Short-lived signed URLs: secure, but expire and should not be stored or surfaced to the model. Generate at click/display time from an authorized trusted service.
• Per-request reference numbers: simple for generation, but merging results or reordering after assignment can swap citations. Freeze the manifest before calling the model.
• Constrained JSON: can prevent arbitrary URL fields and malformed output but cannot prove that an allowed reference actually supports the claim; add span-level grounding validation.
• Model-produced markdown: convenient but dangerous if arbitrary links auto-render; use plain text plus separately bound validated citations.
• Link checks: eager network requests add cost and SSRF exposure; use registry metadata and controlled connector checks, not blind fetching of user/model-supplied links.
• Citation strictness: blocking unresolved citations reduces false provenance but can lower visible citation coverage. Monitor rejections separately, and show honest “source unavailable” states.
• Identity vs location: even a canonical original URL may change after reorganization. Registry IDs, provenance and version contracts must remain authoritative.
• Privacy: URL paths or embedded identifiers may reveal confidential file titles. Render only what the user's current permission permits.

7. Prevention, testing and monitoring

• Add adversarial tests where the model invents plausible URLs, swaps reference IDs, uses [1] after rerank reordering, cites a chunk omitted from the final prompt, mixes versions, or requests a source belonging to another tenant.
• Inject document moves, renamed folders, index alias swaps, deleted documents, expired signed links and redirected destinations; verify stable ID provenance and current authorization.
• Build a golden set mapping answer_claim → allowed citation_ref_id → source/document/version/span → expected URL/viewer behavior. Include intentional “no URL available” records.
• Assert zero rendered links from LLM-provided URL strings, 100% resolved links originate from trusted registry entries, and explicit non-success statuses for unsupported or unauthorized references. The *fraction of reference IDs successfully resolved* can still be below 100%.
• Measure fabricated URL exposure rate, verified citation resolution success, wrong-document link rate, outdated-version link rate, link unavailability, authorization violations, citation support accuracy and p95/p99.
• Compare model-visible reference manifest with emitted IDs and server-resolved sources under correlation IDs. Report which boundary first failed: source metadata, prompt manifest, generation, citation validation, resolver or UI.
• Track schema/connector drift and invalidation on source move, ACL update, source delete or restatement; do not rely on old cached document URLs.
• Human-review sampled high-impact claims and source previews to catch a link that is technically valid but evidentially irrelevant. Re-run tests after prompt, model, retriever, connector or citation-renderer changes.

Interview takeaway: Never let an LLM generate source URLs. Give it a request-local list of allowed citation IDs, validate every emitted ID, and deterministically resolve it through an authorized, version-aware source registry. Render only trusted links; treat unsupported links as failures or unlinked provenance. Check citation identity and claim support separately so a plausible URL never becomes false evidence.

Authoritative references

• Azure AI Search, *Query Knowledge Base via API or MCP* (references, doc keys and service-generated citation URLs): https://learn.microsoft.com/en-us/azure/search/agentic-retrieval-how-to-retrieve
• Azure AI Search, *Create an index for agentic retrieval* (citation metadata fields): https://learn.microsoft.com/en-us/azure/search/agentic-retrieval-how-to-create-index
• Azure AI Search, *Agentic retrieval overview* (grounding references and activity): https://learn.microsoft.com/en-us/azure/search/agentic-retrieval-overview
• AWS Bedrock, *Knowledge Bases data source references*: https://docs.aws.amazon.com/bedrock/latest/userguide/kb-test-retrieve-generate.html