Search works well in English but performs poorly for Hindi-English mixed queries. How would you investigate?

Production incident (illustrative): A bilingual support assistant correctly answers the English query “What is the refund deadline after delivery?” but misses the same policy when a customer asks “Delivery ke baad refund kitne din mein milega?” or “डिलीवरी के बाद refund कितने दिन में मिलेगा?”. Instead of the current 14-day refund policy, retrieval surfaces a generic shipping FAQ. Overall Recall@10 looks healthy because the evaluation suite is 90% English. This is a code-switching and cross-script retrieval regression, not evidence that Hindi-speaking users asked the wrong question.

Key principle: Separate language, script, cross-language alignment, lexical analyzer coverage, domain terminology, and pipeline transformations. A model claiming Hindi support does not automatically perform well on Romanized Hindi, spontaneous code-switching, regional spelling, acronyms or business-specific terminology. Diagnose the *first retrieval boundary that fails* using meaning-equivalent, authorization-equivalent query sets.

1. Triage the production failure

• Preserve a consent-appropriate redacted query trace: the raw original string, Unicode code points for suspicious characters, detected languages/scripts (with confidence), text passed to the embedder, any transliteration/translation/rewrite, tokenizer counts, query-model/version/input role, active vector space, BM25 analyzer, fusion and reranking outputs.
• Confirm the expected active policy chunk exists and is authorized for the user's tenant, geography, date and role. Check source document language(s), title, transliterated terms and OCR fidelity. Do not treat a missing or revoked policy as a multilingual model problem.
• Reproduce the *same intent* with controlled variants: English: “What is the refund deadline after delivery?”; Hindi/Devanagari: “डिलीवरी के बाद रिफंड कितने दिनों में मिलेगा?”; Hinglish/Latin: “Delivery ke baad refund kitne din mein milega?”; mixed script: “डिलीवरी के बाद refund kitne din mein milega?”.
• Keep corpus snapshot, user, ACLs, date/region filters, top-k and search configuration fixed. Log each candidate list and score by lexical, dense, fusion and reranker stage. Changes to any of these other factors undermine language attribution.
• Do not translate, normalize away or rewrite a user's negative or precise constraint without checking that its meaning survives. Hindi “नहीं” and Romanized “nahi” are material; a dropped negation can reverse policy eligibility.

2. Isolate the failure boundary step by step

• A. Source and query preservation: inspect Unicode normalization (NFC/NFKC as appropriate), Devanagari combining marks, zero-width characters, punctuation, mixed numeral forms, capitalization and the original Romanized spelling. Validate any OCR and text extraction separately from retrieval.
• B. Language/script routing: inspect language-ID decisions on short code-switched strings. A detector that labels the entire question “English” may choose an English-only encoder or tokenizer, even if Hindi words dominate the intent. Test code-switch-aware routing or a multilingual default; do not rely on one brittle script heuristic.
• C. Embedding model coverage: compare exact document-ID lookup, exact/flat nearest-neighbor rank and ANN candidate inclusion for the same authorized corpus. If exact vector rank is already poor for Hinglish but good for English, HNSW tuning cannot fix deficient semantic alignment.
• D. Correct embedding roles and compatibility: verify document and query vectors use the intended model/revision/embedding dimensions and query-versus-document input types. An English-only encoder may work for English and fail for Hindi; pin a properly benchmarked multilingual model rather than mixing incompatible vector spaces.
• E. Lexical analyzers: use an analyzer introspection API (for example Elasticsearch _analyze) to compare emitted tokens for refund, रिफंड, refund kab, wapas, वापस. Hindi analyzers can normalize/stem Devanagari words, but do not automatically equate all Romanized spellings with Hindi terms or guarantee code-switch understanding.
• F. Transliteration/translation: test a separate transliteration branch (Romanized Hindi to a canonical Hindi representation) or translation-to-English branch. Measure errors on named entities, brand and product names, negatives, numerals, monetary amounts, dates and technical English terms. Do not assume transliteration is a semantics-preserving one-to-one mapping.
• G. Hybrid fusion and reranker: inspect lexical-versus-dense candidate depths and weights, language-aware reranker behavior, score calibration and query rewrites. A relevant Hindi hit may appear at BM25 rank 8 yet vanish when English-heavy dense scores dominate the merged top-5.
• H. Authorization and retrieval output: verify mandatory filters, active versions and citation targets stay identical across language routes. Never relax ACLs, document-age requirements or tenant isolation when a language fallback is activated.

3. Root-cause decision tree

• English works; both Hindi scripts fail in exact dense ranking: multilingual embedding/translation or underlying corpus-language mismatch. Test a supported multilingual encoder plus compatible re-embedding, or a faithful translation route evaluated against labels.
• Devanagari works; Romanized Hinglish fails: transliteration, Romanized spelling variation, tokenization, language-ID, or insufficient Romanized training examples. Add Romanized query coverage and a robust controlled normalization branch.
• Romanized Hinglish works; Devanagari fails: inspect Unicode/OCR handling, Indic analyzer, Devanagari tokenizer and embedding path.
• Dense works but BM25 fails: diagnose analyzer/token mismatch; use tuned Unicode-aware tokenization, Hindi analysis and approved transliteration/synonym mappings without corrupting exact identifiers.
• Each branch finds evidence but merged top-k does not: examine hybrid fusion ranks, candidate limits, language-biased rerankers and duplicate grouping.
• All retrieval branches return correct chunks, answer still wrong: inspect answer-model multilingual comprehension, prompt language and citation use; that is downstream answer generation rather than first-stage retrieval.
• Only one topic or dialect fails: inspect real user lexicon, domain-specific Hinglish expressions, idioms and multilingual source coverage rather than replacing every model.

4. Worked numerical production example

• Construct 300 meaning-equivalent test questions from 100 real support intents: 100 English, 100 Devanagari Hindi and 100 Romanized Hinglish. Each variant has an adjudicated authorized relevant policy chunk; all share the same corpus and security filters.
• In the baseline, relevant chunk appears in top-10 for English 92/100 = 92%, Hindi 64/100 = 64%, Hinglish 46/100 = 46%. Overall macro average across equally sized language groups is (92 + 64 + 46)/3 = 67.3%. A production dashboard weighted 90% toward English could mask the poor Hinglish segment.
• Known failing Hinglish query Delivery ke baad refund kitne din mein milega? has exact-vector rank 47 for the correct document and ANN rank beyond top-10. Its accurate English paraphrase ranks 3 in exact search. This diagnoses semantic query representation, not HNSW graph health.
• A multilingual encoder experiment using a new matching document-vector index improves illustrative Hindi Recall@10 to 82/100 and Hinglish to 72/100 while English holds at 91/100. The model change alone still leaves 28 Hinglish misses.
• Add a constrained transliteration/translation query branch plus lexical search for domain keywords, then fuse bounded candidates and rerank under the original query and mandatory filters. Suppose test outcomes become English 93/100, Hindi 86/100, Hinglish 84/100. Macro average = (93+86+84)/3 = 87.7%, an improvement of 20.4 percentage points versus 67.3%; Hinglish improves 38 points from 46% to 84%.
• Latency rises from p95 140 ms to p95 250 ms in this example, and indexing/storage increases for parallel vector spaces during migration. Approve only if the p95/p99, cost, security and citation-quality gates remain acceptable. These metrics are illustrative, not measured MachinoAI production results.
• For the refund example, test that nahi/“नहीं”, 14 din/“14 दिन”, order identifiers and tenant restrictions remain unchanged. A top-10 retrieval gain is not a successful fix if negation flips or a revoked document is cited.

5. Engineering solution

• Golden evaluation suite: maintain semantic intent groups with English, Devanagari, Romanized Hinglish, mixed script, spelling variants, technical English loanwords, colloquialisms, negations, dates and exact identifiers. Track quality by language/script and intent rather than only aggregate scores.
• Model and index: compare current English-only and verified multilingual query encoders on the same authorized retrieval tasks. If moving models, rebuild document embeddings consistently into a versioned collection/named vector; never send new-model query vectors against an unrelated old-model document space solely because dimensions match.
• Text handling: store original Unicode text and stable source IDs; maintain separate analyzed/normalized fields, never overwrite raw content or invoice/order identifiers with lossy transliteration.
• Lexical search: benchmark Elasticsearch's hindi analyzer and optional ICU analysis/transform components for actual query vocabulary. The ICU plugin supports Unicode normalization and transliteration, but use precisely tested filters and reindex when index-time analyzer changes require it.
• Query branches: for mixed-script queries, test original multilingual dense query, original lexical query and *one* validated alternate script/translation route. Deduplicate authorized candidates and use a multilingual reranker if evidence supports improvement. Use branch budgets and feature flags, not unbounded translation calls.
• Constrained fidelity checks: preserve dates, amounts, negations, IDs, product names, roles, effective versions and ACLs. Compare translated text to an entity/constraint ledger and reject unsafe rewrites; use original query as a final relevance/answer check.
• Rollout: shadow-evaluate, canary by language/script, compare Recall@k, nDCG, correct-citation rate, wrong-policy rate, p95/p99, token/API costs and query rewriting failure. Roll back *encoder, index alias, analyzer and fusion settings as a consistent bundle* when gates fail.
• Observability: log protected per-stage labels and candidate IDs, analyzer tokens, query-language confidence, model route, translation error class, vector-space version and final support citations. Sample safely; avoid storing sensitive multilingual text unnecessarily.

6. Production trade-offs and common mistakes

• Multilingual embeddings: a single index can simplify cross-language retrieval but may underperform specialized English models on some topics or fail on Romanized dialectal variants. Evaluate both and account for higher per-query latency and reindex cost.
• Translation to English: leverages strong English corpus search, but can change meaning, especially negation, idioms, product codes and geography. Never use it as an unchecked replacement for the original question.
• Transliteration: helps Romanized-to-Devanagari matching but is ambiguous (kal may mean yesterday or tomorrow depending context) and is not the same as translation.
• Hindi lexical analyzers: improve analysis of Devanagari content but may not recover Romanized Hindi, misspellings or cross-script synonyms without extra mappings.
• Multiple search branches: improve coverage but increase latency, complexity and fusion bias; monitor whether English results crowd out the needed policy.
• Low-resource variety: colloquial spellings and English business terms vary across users; synthetic translations can overestimate real-world code-switching performance. Include native-speaker reviews and actual consented query patterns.
• Over-normalization: can erase phonetic distinctions or corrupt numerals, identifiers, legal names and names of customers. Keep original and normalized representations.
• Security-first routing: a multilingual fallback must not bypass tenant/ACL/effective-date restrictions, even when the original script query returns no results.

7. Prevent recurrence

• Make language/script-specific Recall@10, nDCG, Hit@k, correct-citation rate and zero-unauthorized-result gates part of every embedding, analyzer, OCR, translation, fusion and reranker deployment.
• Monitor English versus Devanagari Hindi versus Romanized/mixed Hinglish separately, including fallback rate, zero-result rate, p95/p99, query-language misclassification and constraint-loss incidents.
• Continuously test minimal meaning pairs: refund kab milega versus refund nahi milega, 14 din versus 40 din, differing product IDs and role/date constraints.
• Run canary queries through both query and index generations; verify model compatibility, update freshness, current ACLs and source citations after any migration.
• Review error clusters with Hindi-English bilingual reviewers; classify failures as representation, analyzer, transliteration, domain terminology, filter, fusion, reranker or generation.
• Document tested fallbacks: prefer faithful original-query retrieval, optional constrained translation/transliteration, clear abstention when evidence remains insufficient, and no cross-tenant disclosure.

Interview takeaway: Treat code-switching as a measurable retrieval-path compatibility problem. Compare meaning-equivalent English/Devanagari/Hinglish queries, inspect exact semantic ranks and analyzed tokens, then repair the first failing model, script, lexical or fusion stage. Deploy a multilingual solution only with validated language-specific recall, latency, meaning preservation and authorization.

Research references

• Cohere, Embed models and language support: https://docs.cohere.com/docs/cohere-embed
• Cohere, Multilingual and cross-lingual semantic search: https://docs.cohere.com/docs/semantic-search-with-cohere
• Elasticsearch, Hindi language analyzer: https://www.elastic.co/docs/reference/text-analysis/analysis-lang-analyzer
• Elasticsearch, ICU analysis plugin: https://www.elastic.co/docs/reference/elasticsearch/plugins/analysis-icu
• Elasticsearch, ICU transform token filter: https://www.elastic.co/docs/reference/elasticsearch/plugins/analysis-icu-transform