A user searches for an exact invoice number, but semantic search returns unrelated invoices. How would you debug the failure?

Production incident (illustrative): A finance user asks a RAG assistant, “Show invoice INV-2026-004281.” The correct invoice is present in the document system, but dense-vector search returns several invoices about the same supplier and payment terms. The assistant cites invoice INV-2026-004218 instead—one transposed pair of digits creates a material accounting error. The system has low query latency and healthy vector nodes; the failure is retrieval intent and identifier fidelity, not infrastructure availability.

Key principle: Invoice numbers, purchase-order IDs, serial numbers and account references behave as exact identifiers, not semantic concepts. An embedding model learns similarity of surrounding language; it does not promise exact character equality. Treat a confidently recognized ID as a structured lookup key, with mandatory authorization and version checks, before using semantic retrieval for explanatory questions.

1. Contain the incident and reproduce it

• Prevent the assistant from answering an exact-invoice request using a different invoice merely because its semantic similarity is high. If the exact authorized record cannot be found, return “I could not find that invoice” or request clarification—never substitute a close identifier.
• Capture the raw user request, parsed identifier, tenant/organization, active period, exact ID string, retrieved candidates and scores, vector/lexical query, reranker outputs and the final citation. Redact invoice details in logs.
• Check the authoritative finance system by tenant_id + invoice_number (plus issuer/business-unit when required). Verify whether the requested invoice actually exists, is current, and is visible to the user.
• Reproduce the failure at each boundary: source → OCR/extraction → chunk metadata → keyword/exact index → query parser → retrieval candidate merge → reranker → final citation.
• Preserve raw and normalized representations so debugging does not accidentally hide lost zeros, dashes, spaces, Unicode variants or OCR substitutions.

2. Root-cause debugging decision tree

• ID missing from source or user lacks access: report not found/unauthorized in an appropriately non-disclosing way. Do not search other tenants or weaken ACLs to return “something similar.”
• ID in source but missing from index: inspect OCR errors, CSV/PDF ingestion, page headers, metadata parsing, source version, refresh/replica visibility, and indexing failures. Verify the actual stored field and correct collection/alias.
• ID indexed but keyword/exact lookup fails: inspect mapping and analyzers. On analyzed full-text fields, punctuation may be split or removed and case folded; store a dedicated keyword or normalized exact field. Elasticsearch recommends term queries on exact fields, not analyzed text.
• Exact lookup succeeds but vector query fails: this is expected for many identifiers; measure dense retrieval recall on an identifier-specific evaluation set instead of assuming semantic search must match digits.
• Both retrieve the record but final top-k excludes it: inspect BM25/vector score normalization, reciprocal rank fusion, weighting, per-branch candidate limits, reranker inputs, threshold pruning, duplicate collapsing and query rewriting that drops the identifier.
• Reranker or LLM changes the number: compare exact candidate ID at each stage and require post-retrieval invariant returned_invoice_number == requested_invoice_number for exact lookup requests. Never let generative reasoning replace a structured identity check.
• Unexpected similar invoice appears: test OCR confusables 0/O, 1/I, 5/S, transposed digits, leading-zero stripping, Unicode hyphens, slash normalization, multi-company ID collisions and old invoice versions.

3. Correct retrieval design

• Detect likely exact-ID intent with a constrained parser (regex/validated schema) or a hybrid classifier, while supporting natural requests such as “What was the tax on invoice INV-2026-004281?” Preserve the literal token and avoid LLM rewriting of identifier bytes.
• Canonicalize only known equivalences approved by the source system: e.g., trim surrounding whitespace, normalize known Unicode dash variants and apply documented case rules. Never drop leading zeros or remove arbitrary punctuation if those characters are significant in the issuing system.
• Index both invoice_number_raw and an explicit invoice_number_normalized as exact-match fields. Use composite keys such as tenant_id + issuer_id + normalized_invoice_number; add document version, active status, ACL principal and source lineage.
• Route confidently parsed identifiers to an authoritative database or indexed term/keyword query, with tenant and ACL filtering applied inside the lookup. If exactly one permitted active invoice matches, return it with the authoritative citation.
• For requests with both a valid ID and a conceptual condition (“show INV-2026-004281 and explain why it was delayed”), retrieve the exact invoice first, then use semantic/keyword retrieval *within its authorized related context* for explanation. The exact invoice ID remains a hard constraint.
• If no exact match is found, optionally offer clearly labelled suggestions from fuzzy lookup or trigram similarity, but never silently auto-select a neighboring invoice. Fuzzy suggestions must preserve tenant/ACL constraints and may expose existence only according to policy.
• Use hybrid BM25 + vector search for questions without an unambiguous exact ID, or when users combine a broad topic with multiple terms. OpenSearch supports lexical/semantic hybrid search with normalization or rank-based fusion; tune on an evaluation set rather than assuming a uniform score scale.
• Require output validation and source grounding: the invoice number in a citation, fetched record and final answer must match the requested canonical key. If the invoice is absent or ambiguous, abstain safely.

4. Numerical walkthrough: why the error happens

• A test set has 1,000 invoice-number queries with exactly one existing, authorized active invoice each (separate from tests for missing/ambiguous IDs).
• Dense semantic top-5 returns the exact invoice for only 580 requests: Exact-ID Hit@5 = 580/1,000 = 58%. Another 420 return invoices with similar prose, payment terms or customer descriptions.
• In the illustrative failure, query INV-2026-004281 produces INV-2026-004218 at vector rank 1 with similarity 0.89, while the exact record appears at vector rank 9 with score 0.78. A top-5 cut discards the correct record. The higher embedding score is not evidence that the identifier is equal.
• A dedicated exact lookup using tenant_id=ACME, issuer_id=SUP-17 and invoice_number_normalized=INV-2026-004281 retrieves the verified record in 24 ms in this example. Its identity comes from equality and source fields, not a score threshold.
• After integrating exact routing, suppose 990 of 1,000 test queries succeed. The remaining 10 fail because 6 invoices were stored with OCR-corrupted numbers and 4 have missing canonical metadata. Exact-ID Hit@1 = 990/1,000 = 99%. Repairing ingestion/metadata and retesting those 10 cases is required; do not proclaim 100% coverage prematurely.
• Measured improvement in this hypothetical test is 41 percentage points from 58% to 99% on the relevant success metric, noting the original was top-5 and the new is top-1. For apples-to-apples reporting, separately track baseline Hit@1, post-fix Hit@1, and end-to-end correct-citation rate rather than comparing different k-values as though identical.
• Test realistic negative queries, such as a nonexistent number or an invoice from another tenant. The required result is zero unauthorized disclosure and no silently substituted invoice—not maximal result count.

5. Engineering implementation and validation

• Schema: maintain invoice_number_raw, invoice_number_normalized, tenant_id, issuer_id, source_record_id, document_version, active, acl_policy and content/source hashes. Store identifiers in both document metadata and a queryable exact lookup index.
• Search path: run detect_identifier → validate/canonicalize → authorized term lookup → verify version and source → optional related-document retrieval → citation equality assertion. Keep general dense/hybrid search as a separate route.
• Elasticsearch/OpenSearch example: map invoice numbers to a keyword exact field (with a *deliberately chosen* normalizer if needed) and use a term query under tenant filters. A match query on analyzed text is not a safe equality test.
• PostgreSQL example: use a tenant-scoped indexed equality predicate on the canonical text column; if offering suggestions, pg_trgm supports similarity/trigram searches but does not replace exact matching. Its trigram extraction may ignore punctuation, another reason to preserve exact raw IDs.
• Pipeline tests: inject invoice formats with zeros, dashes, slashes, prefixes, Unicode, OCR confusables, multiple issuers, duplicate revisions and deleted records. Check that normalization does not collapse distinct valid IDs.
• Candidate/rerank tests: assert exact hits survive merge, rerank and context assembly; verify any deduplication is version-aware. A reranker must not “improve” relevance by selecting a neighboring invoice.
• Safe rollout: first shadow-log parser classifications, then canary exact lookups, compare Hit@1, wrong-invoice rate, authorized-not-found rate, p95 latency and citation correctness. Roll back routing rules if false identifier detection harms non-ID queries.

6. Trade-offs and common mistakes

• Exact-only is the safest response for confirmed IDs, but it does not handle vague questions, misspellings or semantic questions without identifiers; use hybrid paths for those intents.
• Strict regex reduces false positives but misses new vendor formats; configurable per-issuer schemas, validation and an unknown-format path are safer than one global pattern.
• Aggressive normalization increases match rate but risks collisions between distinct IDs. Treat normalization as a domain rule, not generic lowercasing and punctuation removal.
• Fuzzy matching helps typo recovery but is dangerous if silently used for financial identifiers. Offer suggestions with visible differences and require user selection.
• BM25/hybrid fusion increases broad recall but can still demote an exact invoice when rank fusion is used blindly; exact route or hard-filter identity first.
• Extra structured index costs storage and synchronization effort; it is justified when identifiers are business-critical and correctness matters more than semantic ranking alone.
• Authorization: a “not found” response may need to avoid revealing whether another tenant's invoice exists. Do not debug by bypassing production access controls.
• Document chunks: invoice IDs buried on page 1 may be absent from chunks of page 3. Preserve parent metadata and structured keys across all versioned chunks, with provenance.

7. Prevent recurrence

• Create an identifier-specific regression suite spanning invoice, order, ticket, customer and error-code formats. Track Exact-ID Hit@1, correct-citation rate, wrong-identifier response rate, false identifier detection and tenant leakage.
• Continuously reconcile source IDs with exact-index metadata and chunk lineage; alert on OCR failures, missing normalized fields, duplicate canonical keys and search-visible lag.
• Add checks in the application that output identifiers equal validated requested identifiers. Block responses with mismatches even when the LLM appears confident.
• Monitor retrieval-path selection and fusion/reranker dropouts, plus query-form distributions, latency and authorized-not-found counts by issuer/format.
• Require an exact-ID search canary before each tokenizer, embedding, hybrid-ranking, OCR or schema release, using real-world formats and security-sensitive negative controls.

Interview takeaway: If a user asks for an exact invoice number, first verify the identifier in a structured, authorized exact-match index, then use semantic search only for related explanation. Debug from OCR and metadata through query parsing, candidate fusion and final citations; never let high embedding similarity override exact identity.

References

• Elasticsearch, term query and analyzed-text warning: https://www.elastic.co/docs/reference/query-languages/query-dsl/query-dsl-term-query
• OpenSearch, hybrid search and rank/score fusion: https://docs.opensearch.org/latest/vector-search/ai-search/hybrid-search/
• OpenSearch, keyword versus vector search: https://docs.opensearch.org/latest/search-plugins/
• PostgreSQL, pg_trgm and punctuation behavior: https://www.postgresql.org/docs/current/pgtrgm.html