Search for technical product codes works during testing but fails for new code formats introduced in production. How would you prevent this regression?

Production incident (illustrative): An enterprise hardware-support RAG assistant reliably finds manuals for product codes like AX-4200 in pre-release tests. A manufacturer then launches codes AX/42.00-R2, AX_4200-EU and 00-AX-4200. Users searching those exact strings get unrelated installation guides or zero results. The index is healthy, but an identifier-extraction regex allows only uppercase letters, digits and hyphens, a text analyzer splits punctuation, a normalizer strips leading zeros, and the offline test suite contains only older formats. A production search-quality regression arises from identifier schema drift, not from a general failure of vector similarity.

Key principle: A product code is a structured identifier with a versioned grammar and an authoritative issuer. Semantic embeddings may help interpret associated product questions, but neither dense similarity nor general linguistic tokenization guarantees exact character fidelity. For exact code intent, correctness requires a source-validated, tenant-authorized identity match; optional prefix/partial/fuzzy search must be a separately labelled fallback.

1. Contain the incident and confirm ground truth

• Identify affected code families, release dates, organizations, regional variants, known missing codes and user requests. Record full input/query path with redacted request_id, exact query bytes, Unicode normalization, parser version, index alias/generation and candidate/citation IDs.
• Check the authoritative product catalog first: does AX/42.00-R2 exist, is it active for the requested region, is the account authorized, and which documents/versions should be linked? A nonexistent or inaccessible product must not be silently replaced with a nearby code.
• Reproduce source → extraction/OCR → canonical identifier → metadata field → indexed exact key → parser → exact term lookup → hybrid branch → reranking → citation. Locate the first failing boundary and keep all stages observable.
• Confirm whether user requests are literal code lookups (“manual for AX/42.00-R2”) or general troubleshooting (“AX 4200 overheating after upgrade”). In both cases preserve an explicit product identifier when present.
• If incorrect manuals are unsafe, disable nearest-code substitution, require exact authorized catalog identity, and present a clear not-found or clarification response until the mapping is repaired.

2. Root-cause decision tree

• New code absent in source: check product catalog launch synchronization and allowed-release/region constraints. Delay retrieval publication rather than manufacturing a record.
• Present in source, absent from search metadata: compare ingestion schema and event versions, OCR extraction, leading zeros, punctuation, long code truncation, malformed Unicode, source aliases, failed bulk items and reindex/refresh visibility.
• Metadata stored but exact lookup fails: inspect the *actual* indexed field mapping and query. Elasticsearch recommends keyword for product IDs, while term is unsuitable on analyzed text fields. Watch ignore_above settings that can omit long keyword values from indexing.
• Parser fails to recognize new format: inspect anchored regex, whitelist, version gates, token boundaries and fallback classifier. A previously safe pattern such as [A-Z]+-[0-9]+ rejects slashes, periods, underscores and leading numeric segments; a malformed detector may incorrectly route the entire query to vector similarity.
• Canonicalization causes collisions: avoid unconditionally removing leading zeros, underscores, hyphens, slashes and periods. AX-4200, AX/42.00 and AX4200 are not automatically equivalent. Equivalence is issuer/catalog-specific and must be versioned.
• Exact lookup works; hybrid result misses: trace lexical candidate list, dense list, candidate cutoffs, RRF/weighting, deduplication, reranker and context assembly. A verified exact match for exact-ID intent is a hard identity constraint, not a score bonus that a different product can overrule.
• Only new formats fail across regions: compare product taxonomy, locale, issuer and entity aliases, document publication delay, query parser model rollout and tenant routing. Multiple manufacturers can legitimately reuse an identical visible code.
• Partial code queries fail: test prefix, authorized field-based wildcards or n-gram strategies separately. Prefix search and substring matches are different user intents with different ambiguity and cost.

3. Map codes as typed, versioned product identities

• Maintain product_code_raw, product_code_canonical, issuer_id, region, product_family, format_version, source_product_id, active_version, tenant_id and ACL/version metadata. Preserve raw exact bytes or a clearly documented Unicode-preserving representation for auditing.
• Use an explicit keyword/exact field for equality; keep a separately analyzed text field for descriptive prose. Elasticsearch's keyword field is intended for structured IDs. Its term-level lookup differs from a match query on tokenized text.
• Store verified manufacturer/catalog aliases as a many-to-one controlled mapping only when the issuer confirms equivalence, with effective dates and collision tests. Don't let regex normalization invent aliases.
• Where product codes need suffix/prefix or infix search, design a separate authorized prefix/wildcard/n-gram field and measured performance bounds. OpenSearch offers a wildcard type for substring/regex scenarios, which should not replace exact product identity.
• Use typed query routing: extract possible code → validate against versioned grammar(s) → authorized catalog lookup → exact index term lookup → confirm source version → optionally retrieve matching product documentation → verify citations.
• If new syntax is unknown, keep the entire literal token, attempt an authorized exact lookup before assuming it is natural language, and emit a structured unknown-format signal for monitoring. A regex is an optimization, not a hard gate to catalog lookup.
• Explicitly version mappings/dynamic templates. Elasticsearch dynamic mapping can infer surprising field types; dynamic=false can silently ignore unknown searchable fields, while dynamic=strict rejects such documents. Either failure should be observable.

4. Numerical worked example

• A pre-release benchmark has 1,000 older-format codes and reports 990 correct top-1 exact product identifications, so Hit@1 = 99%. This figure says nothing about new syntax absent from testing.
• After launch, production includes 700 old-format and 300 new-format product-code queries. Old-format Hit@1 remains 693/700 = 99%, but new-format Hit@1 is 90/300 = 30% because only the fallback lexical/dense path finds some of the new codes. Overall is (693+90)/1000 = 783/1000 = 78.3%, even though the old evaluation suite still reports 99%.
• Controlled debug cases: AX-4200 parses and looks up correctly; AX/42.00-R2 fails the old regex; 00-AX-4200 is normalized to AX-4200 and incorrectly returns a different record; AX_4200-EU is tokenized into fragments in the descriptive text field.
• Deploy a catalog-backed grammar version and dedicated exact keys. In an illustrative retest on the same authorized 1,000-query set, old-format success remains 693/700, and new-format success rises to 288/300 = 96%. Overall becomes 981/1000 = 98.1%, a 19.8 percentage-point improvement versus the 78.3% broken production baseline.
• Remaining 12 new-format misses are classified as 5 ingestion/refresh delays, 4 unknown issuer aliases, and 3 OCR/canonicalization defects. Do not call the upgrade fully correct until each failure class is repaired or safely reported.
• Production p95 may rise from 80 ms to 105 ms due to the authoritative lookup, an illustrative 25 ms cost. Validate p95/p99 against the business SLO. The correctness gate is zero wrong-product substitution and zero cross-tenant disclosure, not simply a high candidate count.

5. Engineering solution: prevent future format changes from breaking retrieval

• Contract with product data owners: ingest a versioned catalog of known formats, sample codes, intended case rules, issuer namespaces, valid character sets and product aliases. Treat format additions as schema changes requiring integration tests and monitoring.
• Dual representation: retain exact literal product IDs and canonical validated keys independently of document embeddings, plus descriptive text for BM25/dense retrieval. Attach reliable product keys to *all related chunks*, not just a title or the first page.
• Parser resilience: use issuer-specific grammars and unknown-format fallback to raw exact catalog lookup; never discard a token because it fails the old regex. Maintain an audit reason code (regex_unknown, catalog_not_found, indexed_missing, ambiguous).
• Index lifecycle: validate field mapping via _mapping and analyzer output via _analyze; create/reindex a versioned exact-field index when mapping/analyzer changes require it. Verify current active products, deleted/retired status and ACLs before alias switch. Do not blindly re-embed unchanged descriptive content when only product-code metadata is broken.
• Batch reconciliation: compare catalog active IDs against exact-index IDs and related document/chunk lineage. Repair partial/failed writes and collect per-issuer visibility lag. A zero-result vector search is not evidence the product is missing from the authoritative catalog.
• Safe partial match UX: if the user supplies an incomplete code, present authorized candidates as suggestions with their full codes and model/region labels; require explicit user selection if ambiguous. Fuzzy candidate similarity must never silently change an exact product ID.
• Canary rollout: shadow test new format families before launch, then canary query routing by issuer/format; monitor exact hit@1, wrong-product rate, unknown-parser rate, authorization failures, p95 and publication lag. Keep an audited rollback, retaining security fixes and current source deletions.

6. Trade-offs and failure scenarios

• Exact matching: reliable for known complete IDs, but strict when users mistype characters. Add a separate transparent suggestions workflow rather than weakening identity.
• Normalization: helps legitimate aliases but may collapse distinct model revisions, locales or manufacturer namespaces. Preserve leading zeros unless catalog rules explicitly state otherwise.
• Wildcard/substring indexing: assists partial lookup at extra storage and query cost; use bounded, tenant-filtered queries and validate high-cardinality performance.
• Regex-only detection: fast for known formats but brittle when suppliers introduce punctuation or new prefixes. Prefer a catalog-driven parser with monitored unknown-format paths.
• Semantic fallback: supports troubleshooting language but can return plausible manuals for the wrong model. Do not let semantic scores override verified structured identities.
• Dynamic mappings: reduce upfront schema work but may infer incorrect data types or ignore new fields under certain configurations. Explicit mappings provide stronger production contracts.
• Full reindexing: may be necessary for mapping changes, but causes temporary storage, ingestion and I/O pressure; use staged migration and lineage reconciliation.
• Cross-tenant and regional codes: identical visible product codes may refer to different items or access scopes. Use composite keys, enforce ACLs and avoid disclosing existence outside authorization.
• Code-like input adversarialness: treat user-provided code strings as data, not regex execution, code, SQL or generated filter expressions; validate length and construct parameterized queries.

7. Prevention and monitoring

• Maintain format-aware golden cases for hyphen, slash, dot, underscore, mixed case, leading zeros, revisions, long alphanumeric codes, Unicode dashes, cross-issuer collisions and region suffixes. Include negative tests for nonexistent and unauthorized codes.
• Partition evaluation by known versus newly launched formats, issuer, rollout date, tokenization/analyzer, query intent and region. Monitor exact-ID Hit@1, wrong-code rate, false not-found, unknown-format rate, ambiguous-match count and correct-citation rate.
• Add contract tests to catalog ingestion, OCR/extraction, keyword mappings, query parser, normalization and RAG citation assembly. Run representative future-format fuzz tests using valid characters and delimiter variants without assuming any one generated format is real.
• Alert when newly observed punctuation or code prefixes exceed a baseline, when parsing falls back to semantic search unexpectedly, or when source active IDs are absent from the exact index.
• Use a release handshake: product-data owner registers new grammar, ingestion canary verifies source/metadata, search canary verifies exact term lookup, and the application checks retrieved product identity before returning cited guidance.
• Review real error samples with product SMEs to catch revisions and aliases missed by regexes; keep source catalog and deletion/ACL policies authoritative.

Interview takeaway: A product-code regression after launch is usually a format-contract failure: an old parser, normalizer, field mapping or test dataset assumes yesterday's syntax. Preserve literal IDs, use authoritative exact-match lookups, version the grammar, and test unseen formats before deployment. Hybrid and dense search should support explanations, not override product identity.

Authoritative references

• Elasticsearch, Keyword field for structured/product identifiers: https://www.elastic.co/docs/reference/elasticsearch/mapping-reference/keyword
• Elasticsearch, Term query exact value behavior: https://www.elastic.co/docs/reference/query-languages/query-dsl/query-dsl-term-query
• Elasticsearch, Explicit and dynamic mappings: https://www.elastic.co/docs/manage-data/data-store/mapping
• Elasticsearch, Dynamic mapping controls: https://www.elastic.co/docs/reference/elasticsearch/mapping-reference/dynamic
• OpenSearch, Wildcard field for substring/code search: https://docs.opensearch.org/latest/mappings/supported-field-types/wildcard/