After deploying a new embedding model, retrieval quality drops despite successful reindexing. How would you troubleshoot vector compatibility?

Production incident (illustrative): A RAG support assistant upgrades to a new embedding model and reports that all 2,000,000 document chunks were successfully reindexed. Nevertheless, Recall@10 falls from 92% to 61% and the assistant retrieves irrelevant troubleshooting procedures. The upsert dashboard is green. Reindex completion proves that vectors were written; it does not prove that queries and documents use a compatible vector space or that the new retrieval pipeline is correct.

Core rule: A vector's dimension is a schema constraint, not a guarantee of semantic compatibility. Two unrelated 768-dimensional embedding models can produce numbers of the same length in entirely different coordinate systems. Query embeddings must be produced using the *matching model family/version and intended query role* for the indexed document embeddings.

1. Contain the incident and establish a comparable baseline

• Stop broad rollout; route a controlled cohort back to the last verified old-model index + old query encoder + old preprocessing, not merely to the old collection while continuing to send new-model query vectors.
• Capture the embedding provider, exact model/revision, dimensions, distance metric, normalization, tokenization, query/document input roles, text templates, truncation, quantization, chunking pipeline version, index/alias, tenant and ACL filters for both ingestion and online queries.
• Collect failing query traces: the user question, transformed query text, query-encoder identifier, vector dimension and norm, target index/vector name, retrieved IDs, scores, filters, rerank scores and citations. Redact private content in telemetry.
• Re-run identical labelled queries through old and new complete pipelines on a consistent eligible corpus, with fixed top-k, filters and snapshot. Separate end-to-end relevance regression from ANN approximation, missing vectors or document version differences.
• Compare retrieved source versions and ACLs; a migration must not reactivate deleted, expired or unauthorized records even if the old index appears to return better results.

2. Root-cause debugging: validate the vector contract

• Model mismatch: query service still calls model A while documents use model B; check endpoint aliases, environment variables, background worker image, cache keys and fallback encoders. Matching dimensions can hide this bug because the database accepts the query.
• Role mismatch: some embedding providers use asymmetric input modes or query/document prefixes. For example, Cohere documents input_type="search_query" for queries and input_type="search_document" for indexed documents. Swapping these or using the document role for both can reduce ranking quality even with one model ID.
• Normalization/metric mismatch: compare cosine, inner-product and Euclidean configurations and the embedding model's intended behavior. Cosine and dot product can rank differently for non-unit vectors; inspect observed vector norms, zero vectors, NaNs, clipping and transforms rather than assuming every provider produces unit-length output.
• Preprocessing mismatch: old index may embed titles + body + metadata while new index embeds only body; query-side prefixes, language selection, whitespace processing, OCR, chunk boundaries, truncation at the provider limit and document templates may have changed.
• Wrong vector target/index: verify collection alias, named-vector target, namespace, tenant, embedding output-dimension setting and index version. Weaviate supports independently configured named vector spaces; queries must select the intended target vector.
• Partial or stale content: reconcile all active IDs/hashes/model-version metadata with new index and check per-shard/index readiness. Successful write counters can mask failed individual items, overwritten IDs or unexpectedly reused cached old-model vectors.
• Ranking and filters: compare dense-only retrieval before hybrid fusion, reranking, metadata eligibility and LLM generation. Changing fusion weights or a reranker during migration can produce apparent "embedding" regression.
• ANN quality: compare exact brute-force nearest neighbors versus approximate HNSW results for the same new-model vectors and filters; if exact quality is good but ANN diverges, investigate ANN build/search settings rather than embeddings.

3. Reproduce with a small compatibility test matrix

• A query → A documents (old known-good pipeline) is the baseline.
• B query → B documents (new intended pipeline) is the correct candidate.
• A query → B documents and B query → A documents are *negative controls*, not meaningful cross-model evaluations unless the models were explicitly trained to share a compatible space.
• Hold the corpus, filters and labels constant, and separately test the provider's correct search_query versus search_document input modes, preprocessing templates and distance metrics. Never compare raw similarity scores across independent vector spaces as though they share one calibrated scale.
• Use a golden vector fingerprint: for fixed non-sensitive strings, record model ID/revision, input role, dimensions, norm range and reproducible vector checksum/tolerance. Detect unintended provider/model changes before production rollout.

4. Numerical incident and engineering diagnosis

• The old index contains 2,000,000 vectors of 768 dimensions, embedded with model A. The newly deployed index contains 2,000,000 vectors of 768 dimensions, embedded with model B. Equal dimensions mean the client/database raise no dimension error, even though the spaces are incompatible.
• A labelled set of 1,000 questions has a known relevant document. Old production Recall@10 finds that document in 920/1,000 = 92% cases. New production finds it in 610/1,000 = 61%: a 31-percentage-point drop, or 33.7% relative decline.
• A trace review finds the online API is still producing model A query vectors but searching the model B document index. This is the primary incident root cause: wrong query/index pairing, not missing document writes.
• After routing queries through model B's correct query encoder and search-query input role, the same evaluation improves to 900/1,000 = 90%. The remaining 2-percentage-point gap from the old baseline still warrants labelled analysis; matching models does not guarantee a model upgrade improves your domain.
• An illustrative second check shows 100 documents missing their title prefix after re-embedding. Restoring the agreed document template and re-embedding only affected versioned chunks raises the measured set to 925/1,000 = 92.5%. This is a hypothetical result, not a guarantee. Confirm no leak/overlap between training, tuning and evaluation sets.
• Sanity check for metric differences: for vectors q=(1,0), d1=(2,0), d2=(0.8,0.6), dot products are 2 and 0.8; cosine similarities are 1.0 and 0.8. Changing from dot product to cosine can change cross-document ranking when norms differ. This example illustrates why the original metric and normalization contract matter.

5. Engineering solution and safe rollout

• Define an immutable embedding contract per index: model_provider, model_id, revision, dimension, distance_metric, normalization, input_type, preprocessing_version, chunk_template_version, quantization, vector_space_id, and source_manifest. Check compatibility at ingestion and query boundaries.
• Enforce query_vector_space_id == target_index_vector_space_id before search; reject or route incompatible queries rather than hoping that equal dimensions are adequate. Use version-tagged embedding caches and separate model-specific named vectors or collections.
• Pin matching query/document model versions; use documented asymmetric query and document modes. Recreate vectors if the embedding configuration or input template changed; do not reuse old vectors solely because the dimensions match.
• Build green separately from the last verified blue index, validate active IDs and ACLs, and run dense-only + filtered exact-vs-ANN tests and labelled Recall@k/nDCG/answer citation evaluation. Compare performance and cost as well as relevance.
• Run shadow traffic and canaries, then switch both query encoder routing and index alias as one coordinated release/configuration version. A vector-db alias switch alone does not atomically update another service's model endpoint; implement a generation-fenced routing manifest.
• If gates fail, roll back the whole retrieval bundle (encoder, preprocessing, vector-space target, metric and index). Retain current deletion/ACL gates and catch up mutations on blue to avoid exposing revoked content. Reconcile any writes accepted during canary.

6. Trade-offs and failure modes

• Parallel indexes/named vectors allow safe comparisons and rollback but temporarily duplicate vector storage and index memory. Some vendors restrict removal of named vectors; plan long-term cleanup.
• Fresh model may improve benchmark results but be worse for the domain, language, document length or rare technical terms. Use representative human-labelled questions rather than generic similarity demos.
• Normalization and metric changes can affect ranking or index efficiency; rebuilding the database with a different metric is not necessarily required when both metrics are mathematically equivalent for consistently unit-normalized vectors, but verify the engine semantics.
• Caching old query embeddings is cheap but dangerous when model IDs change; include model revision, input role and preprocessing parameters in cache keys.
• Hybrid/BM25 fallback can preserve exact-code and product-name retrieval while the dense migration is evaluated, but don't use it to conceal a major dense-search regression.
• Migration CPU, cost and latency rise with double embedding and dual querying; measure p95/p99 and provider token costs while maintaining freshness and version/ACL safety.
• Score thresholds learned under model A may not transfer to model B; re-calibrate retrieval cutoffs and hybrid fusion with labelled data rather than copying score numbers.

7. Prevent recurrence

• Maintain a versioned embedding compatibility matrix and reject cross-space query routing, including accidental model fallback, provider alias changes and cached vectors from older versions.
• Add deploy-time fingerprint tests for fixed sentences, dimensions, input roles, vector norms, distance-metric config and query/document embedding-version parity.
• Require semantic golden-query benchmarks and regression thresholds for Recall@k, nDCG, MRR, citation support, filter correctness and latency before promotion.
• Audit ingested active-point counts and vector-space version per shard; trace online requests to their encoder and target vector space. Alert on wrong-space attempts and missing model-version metadata.
• Run migration drills with stale caches, mixed-model dual writes, delayed events, rolling API deployments and simultaneous document ACL revocations. Test whole-bundle rollback without reactivating unauthorized content.
• Review model release notes and re-test after provider updates, quantization changes or preprocessing refactors; do not assume stable model names imply identical embeddings forever.

Interview takeaway: A successful reindex says the writes landed, not that the query and documents share a compatible embedding space. First verify encoder/version/role, metric, preprocessing, index routing and exact-vs-ANN results; then repair the mismatched pipeline and deploy the query encoder + vector index as a unit.

References

• Weaviate, Switching vectorizers: https://docs.weaviate.io/weaviate/tutorials/vectorizer-migration
• Weaviate, Named vectors: https://docs.weaviate.io/weaviate/manage-objects/create
• Cohere, Embeddings and input types: https://docs.cohere.com/docs/embeddings
• Weaviate, Search target vectors: https://docs.weaviate.io/weaviate/concepts/search