Retrieval-Augmented Generation

Retrieval-augmented generation connects a generative model to a searchable knowledge source. Instead of relying only on model weights, the system fetches relevant documents and places them into the prompt or context.

Your nightly sync accidentally removes 20% of production embeddings. How would you recover without a full rebuild?

Production incident (illustrative): An enterprise RAG system starts the morning with 1,000,000 active chunk embeddings . A nightly sync incorrectly interprets a temporarily empty source connector page as a mass deletion and removes 200,000 vectors (20%) . Search still responds, but users cannot find evidence from many documents. Re embedding the entire corpus would be costly, slow, and unnecessary. Goal: Contain the bad deletion, identify the exact missing document versions/chunks, recover only those vectors from a trustworthy source, and verify real search quality before declaring the incident resolved. 1. Contain the incident immediately Pause the faulty sync and destructive jobs. Disable automated delete propagation and suspend retries or subsequent batches that might delete more data. Preserve original logs, job IDs, index snapshots, and source manifests. Preserve the remaining good index. Take a provider supported snapshot or backup of the current state where feasible. Do not overwrite the live collection with an older snapshot without understanding what legitimate writes happened after that snapshot. Keep retrieval safe. Mark affected document versions RECOVERING; avoid returning misleading “no evidence exists” for queries known to depend on missing content. If available, route affected requests to an authorized lexical index or validated replica, with clear degraded service handling. Record the change boundary: Nightly sync run ID, source cursor, deletion request IDs, first and last affected timestamps, index generation, tenant IDs, and the deletion selection predicate. This is a partial data loss incident . The first task is to stop the delete machinery, not to launch a full reindex. 2. Diagnose: did embeddings disappear, or did search stop seeing them? 1. Compare counts by tenant, namespace, index generation, source connector, document version, and shard. A 20% fall in collection point count suggests physical/logical deletion; unchanged counts with broken search may mean wrong aliases, filters, index refresh, or permissions. 2. Trace the nightly job's source manifest and delete selection logic. Look for a failed pagination request, misread empty result, expired cursor, null source IDs, incorrect tenant scope, or “not seen this run ⇒ delete” code. 3. Inspect vector database delete acknowledgments and operation logs to distinguish delete requests that were issued from deletes actually applied. 4. Compare the expected chunk ID manifest with the actual active chunk IDs . Do not decide recovery solely from total record counts: counts can match despite the wrong chunks being present. 5. Determine the last good snapshot or replica and its timestamp. Identify newly created/updated documents and legitimate deletions since that point; simply rolling everything back could resurrect deleted or permission revoked material. 6. Check for independent embedding artifacts or a durable embedding cache. If matching vectors exist, restore them directly without invoking the embedding model. 7. Audit access control changes. A restored vector must use the current effective ACL , source version, and deletion tombstones, not stale permissions from an old backup. 3. Build a targeted recovery manifest Maintain an authoritative inventory per expected document version: tenant id, canonical document id, source version, chunk id, content hash embedding model id, vector dimension/normalization, chunker version, and index generation Current source status and deletion tombstone, owner, authorization scope, and expected searchable state. Compute: missing chunk ids = eligible expected chunk ids − actual active chunk ids This set difference must be scoped to the same tenant, document versions, model, and index generation . Exclude intentionally deleted, superseded, or inaccessible versions. Also inspect unexpected extra IDs to catch orphan records. A documented manifest is superior to recomputing the entire corpus or relying on file counts. 4. Recover only affected embeddings 1. Preferred: verified existing replica/embedding cache. Read matching vector payloads and metadata from an unaffected, authorized replica or durable artifact store. Confirm hashes, model version, and dimensions. 2. Snapshot staging: Restore the last good provider snapshot into an isolated temporary collection or cluster , when supported. Extract only the missing eligible vectors into a staging index. Avoid restoring an old snapshot directly over the live index: doing so may lose newer writes or revive valid deletions. 3. Re embed only as a fallback: For missing vectors absent from all trustworthy backups, retrieve the current source content, reuse the original chunker/model configuration, and embed only affected chunks . If the document has changed, process its current approved version instead of resurrecting stale chunks. 4. Idempotent restoration: Upsert using deterministic chunk id and versioned metadata. Use bounded batches and per tenant concurrency limits. Track progress in a durable recovery ledger. 5. Version and ACL fencing: Before making restored chunks searchable, re check the source version, current ACLs, and deletion tombstones. Older recovery jobs must not override newer document states. 6. Validate and activate: Check expected versus restored chunk IDs, payload dimensions, searchable visibility, correct tenant isolation, citations, and representative queries before switching any route or marking document versions healthy. 7. Resume the connector cautiously: Fix the bad delete predicate, replay missed legitimate updates, canary the corrected sync, and enable deletions only after reconciliation passes. Backend specific caveat: A snapshot commonly restores a collection or index , not arbitrarily selected individual vectors into an existing live collection. For example, Qdrant documents collection snapshots and recovery priorities; Elasticsearch supports restoring selected indices and restoring into renamed indices. Use a staged restore + selective upserts pattern when your backend cannot do item level snapshot restores. Snapshot compatibility, aliases, and replicas must be checked for that provider. Source: Qdrant snapshots https://qdrant.tech/documentation/snapshots/ ; Elasticsearch snapshot and restore https://www.elastic.co/docs/deploy manage/tools/snapshot and restore 5. Beginner friendly numerical example Assume: Before incident: 1,000,000 valid chunk vectors Accidentally deleted: 200,000 Surviving: 800,000 Of the missing vectors, 150,000 are available in a compatible embedding artifact backup and 50,000 must be regenerated. Embedding cost for this example: $0.0001 per chunk (illustrative; real providers often price by tokens). Then: Lost fraction: 200,000 / 1,000,000 × 100 = 20%. Direct vector restore: 150,000 vectors, with no new embedding inference. Re embedding: 50,000 chunks at $0.0001 = $5 . Full rebuild alternative: 1,000,000 × $0.0001 = $100. Illustrative embedding charge saving: $95 , excluding backup recovery, storage, vector writes, labor, and validation. Suppose each worker safely restores/upserts 1,000 vectors per minute , and 5 workers can operate without saturating the index: 200,000 / (5 × 1,000) = 40 minutes of theoretical write work. Real recovery time also includes snapshot access, embedding generation, index visibility, and validation, so 40 minutes is not an SLA . The target is 1,000,000 correct, currently eligible vectors , not merely returning to a count of one million; some source documents might have changed after the incident. 6. Root cause fix: make deletions conservative and reversible Don't use missing from one scan as proof of deletion. Require authoritative source tombstones or a completed, validated full manifest, and distinguish access errors/empty pages from confirmed removal. Apply a deletion safety gate: If a sync proposes removing an anomalously large fraction (e.g., 5% in one run, a configurable illustrative threshold), hold the operation for review instead of automatically executing. Two phase deletion: First mark candidates PENDING DELETE/tombstoned, reconcile with source and entitlement checks, then perform physical cleanup after validation and a suitable grace period. Security revocations still need immediate query time denial , even if physical cleanup is delayed. Use versioned, auditable delete events: Include source cursor, tenant, document IDs, reason, and batch size; enforce idempotency and conditional source version checks. Protect backups and the recovery manifest from the same faulty connector or credentials that issue deletes. Test failure modes: Missing pagination page, connector returning zero due to authorization loss, partial sync, clock skew, stale cursors, and accidental tenant wide deletes. 7. Verification and prevention Reconcile exact eligible chunk IDs , document versions, and metadata against the manifest; investigate unexpected extras. Run exact ID, keyword, and semantic search canaries on representative missing and surviving documents, plus negative tests ensuring deleted/restricted documents stay invisible. Monitor deletion rate, anomalous batch sizes, source to index drift, recovery coverage, index alias changes, stale source versions, snapshot age, and time since last successful tested restore. Record affected customer queries, restore decisions, recovery start/end, responsible job IDs, and postmortem actions. Document the actual recovery time and recovery point objectives. Re run the corrected sync in shadow/dry run mode before allowing another destructive batch. Trade offs: A recoverable staged snapshot costs temporary storage and I/O but avoids harming current writes. A delayed delete policy reduces accidental data loss but must be paired with immediate tombstone based access denial. Targeted restoration is more operationally complex than a full rebuild, yet preserves newer valid data and dramatically reduces unnecessary embedding work. Interview takeaway: Freeze destructive sync, diff expected and active chunk IDs, restore only eligible missing vectors from a safe replica/cache/staged snapshot, re embed only what cannot be recovered, and verify current versions plus ACLs before reactivation. A fixed count is not proof of correct recovery. Further reading: Qdrant Snapshots; Qdrant Migration and Recovery; Elasticsearch Restore from Snapshot.

A document is partially indexed before a deployment failure. How would you prevent users from seeing incomplete content?

Production incident (illustrative): A healthcare support assistant serves a 40 page policy document. A deployment starts indexing a new revision expected to produce 120 chunks, but the worker crashes after writing only 75. Users now retrieve sections from the new revision without the exclusions and eligibility rules that appear in the remaining 45 chunks. The assistant confidently gives incomplete advice. The indexing job succeeded for individual writes, but the document version was not fully published . 1. Contain the incident Immediately disable retrieval of the incomplete document version using a server enforced visibility filter or active version manifest. Never rely on the LLM prompt to ignore it. Continue serving the last fully verified document version if its contents and access permissions remain valid. Otherwise return a clear "source temporarily unavailable" response instead of a partial answer. Invalidate retrieval, reranker, answer, and citation caches that could contain the incomplete version. Preserve the failed run for investigation. 2. Trace the root cause Correlate tenant id, stable document id, document version, ingestion run id, deployment identifier and timestamps across ingestion logs and index writes. Compare the parser's expected chunk manifest (IDs, page ranges, content hashes, embedding model version) against acknowledged vector writes, actual query visible vectors, document store objects, and ACL metadata. Check batch failures, retries, queue redelivery, indexing lag and whether partial writes were searchable immediately. Root cause: the pipeline treated successful chunk upserts as equivalent to an atomically published document . It exposed the new version before all required data was complete and query visible. 3. Engineer a publish barrier Keep every new revision in STAGING with immutable versioned chunk IDs such as tenant/doc/version/chunk. Never overwrite active chunks in place. Parse, chunk, embed, and write the complete candidate version. Record a durable expected chunk manifest and idempotency key. Retry only missing or failed chunks. Validate all required components : expected IDs and counts, content hashes, text/vector pairing, page coverage, embedding dimensions, metadata, citation targets, tenant scoping, and latest source ACLs. Validate actual read/search visibility, not just successful API responses. Only after validation, switch active version from the old version to the new version using one transactional compare and swap in a strongly consistent metadata store. Every retrieval request must resolve that pointer and enforce a server side version filter; bind the version for the whole retrieval and citation operation. Mark the candidate ACTIVE only after the pointer commit. Retain the previous complete version for rollback and garbage collect old chunks after a safe retention period. For whole index migrations, blue/green collections with an atomic alias switch can be used where the vendor guarantees it. Weaviate supports atomic alias updates; document level publication still needs a version aware serving layer unless each document has its own separately switchable collection. If a vector backend cannot reliably enforce version filters or read after write visibility, route new versions into isolated collections/namespaces and expose them only through a verified serving gateway. A SQL transaction cannot atomically commit arbitrary external vector writes. 4. Numerical example Version 7 contains 100 valid chunks and remains active. Version 8 expects 120 chunks . The failed deploy writes 75/120 = 62.5% . 45 chunks (37.5%) are missing, so the gate rejects publication; users continue retrieving the 100/100 complete chunks of version 7 . The recovery worker replays the manifest and upserts only the missing 45 chunks, then verifies 120/120 , matching hashes, ACLs and query visibility. A single metadata pointer update changes active version: 7 → 8. New requests now see version 8; requests pinned to version 7 finish consistently. If version 8 fails a subsequent canary, switch the pointer back to version 7. No full re embedding is needed. 5. Trade offs and edge cases Staging costs extra storage and writes , but prevents a broken version from leaking into search. Old version retention uses more space. Version filters avoid full index duplication, but every query path (dense, sparse/hybrid, reranking and citations) must enforce them; a missed path can leak staging chunks. Aliases enable simple whole collection cutovers, but may be expensive for single document changes and cannot magically coordinate an external document store. Counts alone are insufficient : duplicate chunk IDs, missing pages, incorrect text, mismatched vectors or stale permissions can pass a count check. Use a manifest plus integrity and authorization checks. If the old version was revoked or contains sensitive content, fail closed rather than serve it. A deletion or permission revocation takes precedence over availability. Concurrent updates require generation fencing/compare and swap so a late retry cannot republish an older revision. Account for vector index propagation lag and in flight requests. 6. Prevent recurrence Treat ingestion as a state machine: DISCOVERED → STAGING → VALIDATING → READY → ACTIVE, with FAILED and ROLLED BACK paths. Make write/replay steps idempotent and publication conditional on a verified manifest and source version fence. Add deployment fault injection tests: crash after 1%, 50%, and 99% of chunks; delayed indexing; duplicated events; ACL changes; concurrent versions. Alert on stuck staging jobs, expected versus visible chunk mismatch, staging chunks returned by retrieval, failed version swaps and partial citation coverage. Perform post publish canary queries, check citations and permissions, and retain an audit trail of who or what promoted each version. Interview takeaway: Make ingestion incremental but publication atomic from the user's perspective : write privately, validate completeness and security, then atomically change the serving pointer. Never let a successful chunk upsert imply that an entire document revision is ready. References Weaviate: Collection aliases — https://docs.weaviate.io/weaviate/manage collections/collection aliases Weaviate: Zero downtime collection migration — https://docs.weaviate.io/weaviate/tutorials/collection aliases MachinoAI: Updating or deleting indexed RAG documents — https://www.machinoai.in/concepts/rag/how do you update or delete indexed documents in a rag vector database

Your RAG system combines data from APIs updated every minute and PDFs updated monthly. How would you enforce appropriate freshness guarantees?

Production incident (illustrative): A support assistant answers a customer's question about inventory availability and warranty eligibility. Inventory changes every minute through an API; warranty policy PDFs are revised monthly. At 10:07 the assistant says an item has 40 units available, citing an inventory snapshot captured at 09:50, although the API now reports 2 units. It also quotes a warranty policy published three weeks ago, which remains authoritative. Applying the same 24 hour cache TTL to both sources produces a dangerously outdated inventory answer even though the PDF answer is acceptable. Core principle: Freshness is a per source, per field, and per use case contract , not a universal re embedding interval. Distinguish how old a value is , how quickly the source changes , whether it is still authoritative , and what the user needs at answer time . 1. Contain and debug the production incident For transactional or safety critical answers, stop relying on a stale vectorized API snapshot. Fetch from the authoritative live API, or abstain with a clear "current data unavailable" message when it fails. Inspect ingestion logs, API polling/webhook events, queue lag, dead letter queues, index acknowledgment times, search read visibility, and cache hit metadata. Trace tenant id, source id, record version, source updated at, observed at, indexed at, last verified at, and response as of. Establish whether the lag came from an API poll scheduled too slowly, missed change events, throttling, a stuck job, retries, eventually consistent search writes, stale CDN/application caches, or an LLM response cache that ignored source versions. Check source authoritative timestamps instead of assuming indexed at equals data freshness. An index can be rebuilt today from yesterday's API extract. Root cause: a single TTL and a shared ingestion cadence incorrectly treated minute level API state and slow changing reference documents as having identical freshness requirements. 2. Define explicit freshness contracts Live API facts (inventory, pricing, availability): example freshness SLO ≤ 2 minutes at the point of answering, with a ≤ 30 second target for critical order confirmation. These are illustrative business requirements, not universal defaults. PDF policy documents: sync on version/change notifications plus a scheduled reconciliation (for example daily), with a ≤ 24 hour discovery lag goal for revisions, despite normal monthly revision frequency. A PDF being "monthly" does not make every PDF safe to serve for 30 days. For each source and field, define max age, preferred refresh mechanism, allowed stale behavior, error budget, source ownership, revocation priority, and escalation. Distinguish freshness (age since the source's last confirmed state) from completeness (all required updates incorporated), authority (which version governs), and latency (retrieval response time). Track both actual event propagation and periodic successful source verification; unchanged sources should not appear permanently stale solely because no update event arrived. Time sensitive requests ("right now", "can I order?") should force live verification. Historical questions may explicitly use an as of timestamp and versioned snapshot. 3. Engineer two complementary data paths Fast changing API path: ingest authenticated change events/webhooks or CDC where supported; use short interval polling with conditional requests/version tokens as a fallback. Queue events durably, deduplicate by record ID and version, handle out of order messages, and retry with exponential backoff + dead letter queues. Maintain structured current state storage for numeric attributes; use RAG embeddings only for descriptive/searchable context. Slow changing PDF path: watch source revision IDs, hashes, effective dates, ownership and ACL changes; parse/embed only changed document versions; stage and validate a complete revision, then atomically promote the active version. Run reconciliation to catch missed notifications. At query time: classify each retrieved claim by required freshness. For inventory/price, call the live service (or sufficiently fresh authoritative snapshot) after retrieval identifies the entity, then generate a cited answer. For policy, retrieve only the current effective document version. Carry source updated at, verified at, version and as of timestamps through citations. Enforce freshness in code: check age before passing data to the LLM. If the source is too old, refresh synchronously within an explicit timeout; if unsuccessful, fail closed for critical fields. An unrestricted stale while revalidate response is appropriate only for low risk content with a clearly communicated age. Bound answer caches by the shortest relevant source contract, and use source version/etag plus tenant and authorization scope in cache keys. Invalidate proactively on source changes, deletions and permission revocations. 4. Numerical example: does the result meet its SLO? At 10:07:00 , the inventory record used by retrieval has verified at = 09:50:00: 17 minutes old . Its 2 minute max age is exceeded by 15 minutes , so the retrieval middleware rejects it. A live API call returns 2 units with a successful check at 10:07:01 ; the generated answer says " 2 units as of 10:07:01 ". No model generated extrapolation from the previous 40 units is permitted. A warranty PDF was published 21 days ago but the connector last verified its current revision and effective status at 09:00 . Its verification age is 67 minutes , inside an illustrative 24 hour source verification SLO. It remains eligible, provided no revocation/effective date signal contradicts it. In a sample of 10,000 API answer requests , suppose 9,850 used source confirmed values no older than 2 minutes. The freshness compliance rate is 9,850 / 10,000 × 100 = 98.5% . If the target is 99.9% , the system misses by 1.4 percentage points , or 140 additional compliant requests for that sample. Illustrative propagation budget: 20 seconds source event delivery + 25 seconds processing + 10 seconds read visibility + 15 seconds cache propagation = 70 seconds , below 120 seconds. However, delays compound and percentiles must be measured end to end; component averages do not prove the freshness SLO. 5. Trade offs and tricky cases Live API calls: strongest currency for critical facts but add latency, per request cost, rate limit dependence, and downstream outage exposure. Event driven ingestion: low normal lag but missed events and out of order delivery require replay, sequence/version fencing and periodic reconciliation. Event time watermarks help reason about lateness; they do not guarantee that an external API has delivered all updates. Frequent polling: simpler when webhooks are absent, but wasteful and vulnerable to quotas; interval alone cannot guarantee a strict maximum age without an explicit lag allowance. Aggressive TTL: cheaper and faster but can serve stale results. Short TTL: fresher but creates load and thundering herd risk; use request coalescing and bounded jitter. PDF updated monthly: authoring cadence is not a contractual freshness window; security revocations, access changes, emergency bulletins and effective date rules must propagate promptly. Clock skew and unchanged records: prefer trusted source versions and successful last verified at; monitor event to observation lag rather than treating old source updated at as proof of pipeline failure. Cross source contradictions: show individual timestamps, select authoritative owners by field, and avoid merging values collected at incompatible points in time. Critical failures: if source verification times out, say availability is unverified; never silently substitute stale data for order critical numbers. 6. Prevent recurrence Publish a freshness policy registry by source, field and risk class, reviewed with product and data owners. Instrument event age , verification age , ingestion lag , index visibility lag , cache staleness , query time rejection rate , and freshness SLO compliance with per source percentile dashboards. Alert on missing heartbeats, dead letter queue growth, poll failures, missed source version changes, and any unauthorized/expired source offered to the model. Exercise chaos tests for dropped webhook events, delayed queues, clock skew, API 429/5xx, PDF emergency replacement, ACL revocation and cached answers spanning a change. Store source IDs, versions, timestamps and source of truth references in an auditable answer trace; periodically compare answered values with direct source reads. Interview takeaway: Don't choose a single TTL for every RAG source. Use freshness budgets based on the business meaning of each fact : event driven ingestion plus live verification for high churn API state, version aware incremental indexing for PDFs, and a retrieval time freshness gate that abstains when a critical source cannot be verified. Further reading Confluent, Time and Watermarks in Flink: https://docs.confluent.io/cloud/current/flink/concepts/timely stream processing.html Confluent, Handle Late Arriving Data: https://docs.confluent.io/cloud/current/flink/how to guides/handle late arriving data.html AWS, EventBridge Scheduler retry and dead letter queues: https://docs.aws.amazon.com/scheduler/latest/UserGuide/managing schedule.html

Two source systems disagree about which document version is authoritative. How would you resolve ownership and precedence?

Production incident (illustrative): A benefits RAG assistant answers that employees receive 20 days of parental leave , citing an intranet PDF. The HR policy management system says 30 days under an approved revision. Both versions are indexed, both appear current, and the older intranet file ranks higher because its wording closely matches the question. The assistant repeatedly gives the wrong entitlement. This is not fundamentally an embedding problem: two publishing systems are claiming authority over the same policy. 1. Contain and scope the disagreement Flag the affected canonical policy id and policy scope (country, legal entity, workforce type, effective date). Temporarily block unsupported answers on this policy or route them to human approved guidance; do not silently pick the top ranked snippet. Capture the exact question, response, citation URLs, candidate source IDs, hashes, approval states, source timestamps, ingestion run IDs, tenant and ACLs. Notify the accountable business policy owner and ingestion owners. Preserve both records and audit evidence; don't delete conflicting copies before resolving lineage. For legally material or customer impacting decisions, fail closed or explicitly identify the unresolved discrepancy. Authorization revocations take priority over version disputes. 2. Root cause debugging checklist Compare source semantics: is one system the actual system of record , while the other is a syndication mirror, collaboration workspace, draft folder or cached export? Check approved by, approval workflow, owner, policy revision ID, signed/published state, jurisdiction, effective from/to, supersession links, and source system revision sequence. Distinguish event time (when the policy applies), approval time , source last modified , and indexing time . Newest upload or highest embedding similarity is not proof of authority. Follow lineage from original policy through copies/exports to chunks and citations. Look for an outdated mirror, a misconfigured connector marking drafts as ACTIVE, missed change notifications, or conflicting IDs mapped to separate canonical policies. Root cause in this example: both the canonical HR system and a replicated intranet PDF were treated as equal publishers; no enforceable authority registry, owner decision, or precedence gate existed ahead of retrieval. 3. Engineer explicit ownership and precedence Set up a governed source authority registry , keyed by tenant id + policy type + business scope, not a one size fits all global ranking. Store business owner, technical steward, primary source, eligible fallbacks, approval rule, effective date rules, policy version and decision approver. Define authority as business ownership and approved publication rights , confirmed in governance records; don't infer it from last write wins, URL domains, filename or vector scores. A designated system of record may vary by entity/country/policy type. Map duplicates and exports to one stable canonical policy id; maintain separate source record id, source system, source version, content hash and derived from lineage per copy. Preserve the original citations and approval evidence. Build a precedence decision gate , in order: 1. Authenticate tenant and enforce row/document ACLs for every candidate. 2. Determine applicable jurisdiction, entity, population and requested as of date. 3. Require source ownership/approval and non revoked status. Verify that the candidate falls within its valid effective interval. 4. Select the designated primary authority for that policy scope; consult approved fallback sources only if governance explicitly allows it and freshness/consistency checks pass. 5. Resolve within that source using a documented supersession/revision chain. A higher revision number across different sources is not comparable. 6. If equally authorized sources conflict or the declared primary cannot be verified, quarantine the policy for escalation rather than inventing a winner. Apply the authority and effective version eligibility gate before vector similarity ranking . Keep drafts, mirrors and unapproved copies searchable for internal auditing but not eligible for production answers. Use a versioned, signed/approved authority decision record (who, when, evidence, scope, rule version). Publish a new retrieval eligibility manifest or metadata state only after validating references and source ACLs. Invalidate stale retrieval/answer caches, then test representative user queries. If two independent business owners legitimately retain separate rules, do not collapse them into one golden record: partition by scope and ask for missing context when needed. 4. Numerical production example At 09:00 the intranet export POL LEAVE v4 says 20 days , with similarity score 0.94 . The approved HR system POL LEAVE v5 says 30 days , with similarity score 0.82 . Naive top 1 retrieval selects v4 (0.94 0.82) , returning the incorrect 20 day entitlement. Governance registry says HR Policy Manager is primary for India/full time/2026 , with v5 approved and effective 1 October 2026 ; intranet is a read only derivative, last synced 20 September . An authority filter excludes v4 for current answers despite its higher semantic score. The assistant returns 30 days , citing v5, its scope and effective date. For an explicitly historical question dated 1 September 2026 , the correct result may be v4 if its approval and effective window records confirm it . Suppose a test set has 1,000 current scope policy questions . Before the gate, 120 return an unauthorized/outdated version: 120/1,000 = 12% wrong authority rate. After deployment, 8 do: 8/1,000 = 0.8% ; that's an 11.2 percentage point reduction and 93.3% relative reduction in wrong authority answers. The remaining eight require investigation, not an assumption of perfection. 5. Trade offs and failure modes Centralized authority registry improves auditability and predictable answers, but requires ongoing stewardship; an outdated registry can enforce the wrong source at scale. Per scope ownership supports multinational policy differences, but adds metadata complexity and may require clarifying user context. Hard source filters reduce the risk of persuasive but unauthorized answers, though they may reduce recall and increase abstentions when the primary source is unavailable. Approved fallback can improve availability but may resurrect stale guidance. For high risk policies, require explicit owner approval and TTL rather than automatic fallback. Manual arbitration is slower than heuristic recency ranking, but necessary when approval authority is genuinely disputed. Add deadlines and escalation tiers. Golden record approach works when one authorized version exists; it should not flatten legally distinct country rules, future effective policies, or historical answers. Security is separate from document authority: a correct authoritative file can still be forbidden to a given user. Keep tenant/ACL checks mandatory and never cite a restricted source. 6. Prevent recurrence Require one accountable business owner per policy scope , an up to date authority registry, and an explicit exception/appeals workflow for overlapping ownership. Validate canonical IDs, origin chains, approval metadata, effective windows, ACL inheritance and primary source designation during ingestion. Reject unknown authority states from the serving index. Monitor multiple active authority conflicts, mirror lag, missing owner, unapproved source retrieval, authority registry drift, stale citation caches and unresolved dispute age. Add canary tests with higher scoring but unauthorized mirror text , overlapping effective windows, late arriving approvals, disputed owners, revoked ACLs and historical queries. Reconcile the source registry with business owners on a regular cadence and audit every promotion/override with evidence and approver identity. Interview takeaway: When two document systems disagree, first decide who is authorized to publish policy for the applicable scope , then enforce ownership, approval, effective dates and lineage before semantic ranking. Similarity and timestamps decide relevance and freshness, not authority . Further reading Microsoft Purview: Master data management — https://learn.microsoft.com/en us/purview/data governance master data management Microsoft Purview: Data lineage in the Data Catalog — https://learn.microsoft.com/en us/purview/data gov classic lineage

How would you design an auditable document-state machine that tracks ingestion, indexing, publication, supersession, and deletion?

Production incident (illustrative): An enterprise RAG assistant quotes a confidential policy PDF three hours after the source owner deleted it . Logs show the ingestion job finished, a vector delete timed out, and a second worker retried an old publish message. The monitoring dashboard says the document is "processed," but nobody can prove which version was visible at any moment. The incident exposes a governance design flaw: background tasks are not a document lifecycle , and a generic "success" status is not evidence that publication, supersession, or deletion actually completed. 1. Contain and trace the incident Immediately block the affected tenant id + canonical document id at the retrieval authorization/visibility gate . Invalidate answer, retrieval, citation and CDN caches; halt late publish jobs for that document. Preserve the source deletion notice, worker run IDs, event identifiers, vector deletion receipts, active version pointer history and user visible citations. Build a timestamped timeline: source deleted at → deletion requested → serving blocked → vector delete requested → vector delete verified → purge complete. Determine whether the system acknowledged a queue message before persisting its state transition; whether a stale worker republished a tombstoned version; and whether partial cleanup was mistaken for completion. Root cause: mutable status fields and disconnected worker logs had no guarded transitions, durable event audit, or generation fencing. An asynchronous delete and a delayed publish could both claim success. 2. Define the document state machine Track each immutable document version independently, and maintain a separate canonical document record pointing at the currently eligible version. DISCOVERED: source event received; canonical identity, tenant and source revision recorded. FETCHING: source bytes and authenticated metadata are being captured. STAGED: immutable source snapshot, hash, lineage and metadata persisted. PROCESSING: extraction, parsing, chunking and embeddings in progress. INDEXING: version specific chunks are being upserted to a non serving candidate. VALIDATING: verify chunk manifest, search visibility, citation targets, ACLs, completeness, source revision and security. READY: all validation checks pass, but the version is not yet user visible . PUBLISHED: a guarded, atomic serving pointer promotion makes the complete eligible version visible. SUPERSEDED: a newer authorized version has been promoted; the old version is no longer eligible for ordinary current queries but may be retained for permitted historical queries. DELETION REQUESTED: tombstone/deny serving flag committed immediately , overriding any READY/PUBLISHED version. No new query may retrieve its content. DELETING: vector, source object, derivative and cache purges are underway. DELETED: deletion has been verified across the defined serving/storage scope ; record audit evidence and known retention exceptions. Do not equate a successful delete API request with confirmed deletion. FAILED / QUARANTINED: a recoverable failure or integrity/security exception; preserve the last safe published version only if not revoked. Retry via explicit authorized recovery transitions. Allowed transitions, not free form status updates: DISCOVERED → FETCHING → STAGED → PROCESSING → INDEXING → VALIDATING → READY → PUBLISHED → SUPERSEDED. From any non deleted state, a valid deletion command can move the canonical document to DELETION REQUESTED; then DELETING → DELETED. Failures branch to FAILED or QUARANTINED with controlled retry. A superseded version must never automatically become PUBLISHED again; an authorized rollback must create an auditable promotion decision and pass current permission/effective date checks. 3. Persist state and evidence separately Document aggregate: tenant id, canonical document id, active version id, lifecycle generation, serving blocked, deleted at, updated at. Immutable version record: document version id, source system, source record id, source revision, content sha256, manifest sha256, acl policy version, status, expected chunk count, visible chunk count, supersedes version id. Append only transition ledger: event id, aggregate id, version id, generation, from state, to state, event type, actor/service, occurred at, recorded at, workflow run id, correlation id, idempotency key, reason, evidence uri, and a schema version. Transition ledger records support investigation, but true compliance grade immutability may need restricted write privileges, tamper evident hashes/signatures, access logs, immutable retention and independent backups; a writable application table alone is not tamper proof. Use one database transaction to compare the expected state/generation, update state and append its audit event. Add a transactional outbox event in the same transaction; a separate relay dispatches work after commit. Make workers at least once safe : deduplicate by event/idempotency key, use versioned chunk IDs, and fence retries with generation/sequence checks. Out of order older events must not overwrite a newer state. External vector databases and object stores are not in the SQL transaction. Use reconciliation, receipt checks, retry safe compensating actions, and explicit observable DELETING / VALIDATING states instead of claiming cross system atomicity. 4. Publication, supersession and deletion invariants Publish: only READY with verified full chunk manifest, current source authorization, ACLs and document generation can promote the canonical active version pointer. Retrieval must apply the active version filter and current tenant/ACL rules on every access path. Supersede: promote the validated new version and demote the old version in a single guarded metadata operation; preserve citation/source lineage and historical eligibility only where policy permits. Delete: commit the deny serving tombstone before asynchronous erasure. Every retrieval and cache path must respect it, including delayed requests and any replica. Reject later indexing and publish events for the revoked generation. Deletion completion: check vector IDs, text chunks, stored source snapshots, previews, reranker/answer caches and scheduled backfills; document any approved immutable backup or legal hold exception honestly. Recovery: a failed attempt never skips required validation. Replaying history should reconstruct both the last committed state and pending external actions; manual overrides require named approvers and reasons. 5. Beginner friendly numerical incident Document DOC 42, version v1 , has 80 verified chunks and is PUBLISHED. An update creates v2 with 100 expected chunks . After a worker crash, 70 of 100 v2 chunks are indexed: completeness = 70/100 = 70% . Since publication requires 100% of manifest IDs plus integrity/ACL checks, v2 stays INDEXING and is invisible; v1's 80 chunks remain eligible. A retry writes the 30 missing chunks idempotently. Validation confirms 100/100 ; a guarded transaction records v2 READY → PUBLISHED, changes active version = v1 → v2, and records v1 → SUPERSEDED. At 14:00 , source owner deletes DOC 42. The system commits a tombstone by 14:00:02 (2 seconds later); all query paths now reject DOC 42 even though physical vector deletion takes 4 minutes . A stale worker's publish(v1) event is rejected because its generation is old. By 14:04 , deletion reconciliation confirms 180/180 stored vector IDs removed (80 from v1 + 100 from v2), plus verified associated caches/objects within the specified scope. Only then transition to DELETED. Historical events remain in a protected audit ledger with minimal content identifiers according to retention policy. In a batch of 1,000 lifecycle transitions , assume 997 carry valid audit events and evidence links; audit completeness = 99.7% . A target of 100% required events means three missing entries are incidents to investigate, not acceptable silent gaps. This is an illustrative KPI. 6. Trade offs and edge cases DB state + append only events is straightforward for audits, but increases schema and operational complexity. Event sourced workflow engines offer durable histories/replay; still implement authorization and external service reconciliation. Strict gates improve safety but delay publication and raise temporary unavailability. Avoid bypassing validation for urgent changes; offer controlled emergency approval with full audit records. Exactly once end to end is usually unrealistic across a queue, SQL database and vector store. Prefer at least once delivery, idempotent side effects, generation fencing and observable completion. Deletion vs evidence retention: audit metadata may remain, while document content and embeddings must follow contractual/legal deletion requirements. Record retention exceptions and separate "blocked from serving" from "physically purged". Concurrent updates and deletes: deletion wins immediately; rejected late retries are logged. A legitimate restore is a new authorized ingestion/generation , never a resurrection by retry. Clock skew: use database sequence/order for committed transitions; preserve separate source and observed timestamps. Do not determine causal order using wall clock timestamps alone. Workflow history cost: high volume chunk events can explode history size. Record meaningful document transitions with manifest pointers and aggregate counts; store per chunk details in separate partitioned evidence storage. 7. Prevent recurrence Define allowed transitions and invariants as code with guarded compare and swap operations, not free form worker status updates. Test crashes before/after SQL commit, before/after outbox dispatch, halfway through indexing, during version promotion, and during deletion. Inject duplicate, reordered and delayed messages. Alert on READY without verified manifest, PUBLISHED with missing source/ACL evidence, stuck DELETING, tombstoned content returned by any retrieval path, stale generation publishes, or gaps in the transition ledger. Run scheduled reconciliation between source inventory, SQL aggregate/version state, actual vector IDs, object store contents and live serving/citation results. Export a deletion evidence report. Review transition overrides, actor permissions, audit retention policies, outbox backlog and deletion SLAs with security and document owners. Interview takeaway: Treat production RAG ingestion as a guarded state machine with durable transition evidence , not a collection of loosely coordinated jobs. Publication requires a verified, atomically selected version; deletion must immediately block serving and only complete after independent purge verification. References Temporal: Event History — https://docs.temporal.io/encyclopedia/event history AWS Prescriptive Guidance: Transactional outbox pattern — https://docs.aws.amazon.com/prescriptive guidance/latest/cloud design patterns/transactional outbox.html AWS Step Functions: Viewing execution details — https://docs.aws.amazon.com/step functions/latest/dg/concepts view execution details.html

Vector search p99 latency suddenly increases while p50 remains stable. How would you identify the responsible infrastructure bottleneck?

Production incident (illustrative): A customer support RAG service queries 8 million product document vectors. After the nightly ingestion batch grows, vector search p50 stays at 45 ms , but p99 jumps from 180 ms to 1,250 ms . Most users still see fast responses; a small minority experiences timeouts. Raising the LLM timeout hides the symptom and doesn't identify the vector search bottleneck. Key distinction: p50 is the median request, whereas p99 is the latency at or below which approximately 99% of requests fall in a chosen observation window. Rising p99 with stable p50 suggests a slow subset, not proof of a specific cause . Investigate query shape, tenant, shard, node, queueing and storage behavior before changing index settings. 1. Contain and establish the boundary Check the customer facing latency SLO, timeout rate, affected tenants and rate of bad requests; if necessary throttle bulk ingestion, bound request concurrency or shift traffic to a healthy replica through a controlled rollout. Confirm whether the alert measures vector DB server time , client observed search time , or end to end RAG time . Record these separately; reranking, query embedding and LLM generation can affect end to end p99 even if vector search is healthy. Compare the same metric, route, traffic mix, time window, region and percentile aggregation with the previous baseline. Do not average p99 values computed across shards or time slices; use raw distributions/histogram aggregation where valid. Inspect changes to ingestion jobs, index settings, top k, search ef, payload filters, quantization, shard placement, autoscaling, storage tier, deployment versions and traffic bursts. 2. Trace the actual slow requests Add a trace/correlation ID across API ingress → pool/queue wait → vector client network → coordinator → scatter/gather → shard search → filter evaluation → HNSW traversal/rescore → payload fetch → response serialization. Log request size, top k, ef, filter selectivity, collection, tenant, shard and target replica without leaking sensitive query text. Compare slow top 1% against normal requests on the same route, including a request sampled profile, not just global CPU averages. Look at DB native profiling where available. Weaviate, for example, exposes per shard query profiling fields such as vector search took, filter allow list construction, and HNSW traversal/rescore timings. Use the provider's actual supported telemetry rather than assuming these field names apply across vendors. Measure queue wait and concurrency separately from execution. A saturated client connection pool can inflate client observed p99 even when server execution stays low. 3. Classify the responsible bottleneck Storage and cache misses: slow traces correlate with read I/O spikes, storage await, page faults, cache evictions or cold on disk HNSW/payload access. Qdrant warns that colder index/payload tiers can increase disk I/O and latency. Compare repeated warm queries with first time cold queries. CPU or background index contention: slow windows align with bulk upserts, segment compaction/optimization, HNSW rebuilds, CPU run queues or CPU throttling. Search and optimizer workers may compete for CPU, RAM and I/O. Stop or rate limit the batch as a reversible experiment. Query planner/filter issue: slow requests use selective or unindexed metadata filters, unusually large top k/offset, costly hybrid search or changed ANN search effort. Inspect whether the filter uses an index, builds an expensive candidate list, or triggers a full scan. Do not simply lower search effort before measuring recall. Network/coordination tail: one straggling replica, transient retries, cross zone traffic, replica read inconsistency, unavailable shards, or coordinator fan out magnifies p99. Inspect per shard/peer traces and retry counts; a minority of slow shards is sufficient to hurt the slowest queries. Load imbalance and queueing: bursts concentrate on a tenant or shard, connection pool wait rises, or a load balancer sends disproportionate traffic to one replica. Check per node request rates, queue depths and utilization, not cluster averages. 4. Practical numerical root cause walkthrough Over 10,000 comparable searches in a five minute window, assume 9,800 finish near 45–80 ms while 200 filtered searches suffer significant delays. The slow 2% is sufficient to move p99 dramatically while p50 remains about 45 ms. Before incident: p50 = 45 ms , p99 = 180 ms . During incident: p50 = 46 ms , p99 = 1,250 ms . Tail inflation = 1,250 / 180 ≈ 6.94× , despite only 1 ms median increase. In a representative slow request trace: API/queue 20 ms + coordinator/network 25 ms + filter planning 90 ms + vector search and payload I/O 1,070 ms + serialization 45 ms = 1,250 ms . This is a single illustrative trace , not a decomposition of the aggregated p99. Node metrics show a spike in disk read latency and vector I/O on one replica during heavy index optimization. With the ingestion batch paused for a short controlled test, representative slow request vector I/O falls from 1,070 ms to 110 ms , while network and filter time remain similar. This correlation plus the isolated intervention implicates storage contention from ingestion/optimization , not the LLM or semantic search score. Engineering fix: throttle indexing and cap concurrent writes, move hot index working sets to a suitable faster memory/storage tier if justified, and schedule expensive optimization work appropriately. Validate with the same replay traffic; suppose p99 falls to 210 ms and p50 remains 46 ms . Tail improves (1,250−210)/1,250 × 100 = 83.2% , though it has not returned to the 180 ms baseline. 5. Make the change safely Choose the smallest reversible mitigation backed by traces: cap ingestion concurrency, rebalance overloaded replicas, tune storage tiers or optimize a missing payload index. Treat index rebuilds or migrations as controlled operations. Re run a representative benchmark with actual metadata filters, tenant mix, burst rate, cold starts, and write workload. Compare p50/p95/p99, throughput, failure rate, queue wait, CPU throttling, disk IOPS/latency and per shard results. For changes to HNSW ef, quantization, top k or filter strategy, simultaneously check Recall@k and correctness ; a faster result can be worse retrieval. Watch query time access control and tenant filtering. Roll out through canaries, with rollback thresholds; preserve a before/after trace set and configuration diff to confirm causality. 6. Trade offs and edge cases More RAM / pinned index: fewer cache misses, higher cost; do not scale first if queueing or a bad filter dominates. Faster SSD / more IOPS: helps I/O bound tails, but may do little for CPU or lock contention. Limit write concurrency: protects search p99 but increases ingestion to search freshness lag; set a freshness budget and reserve ingestion capacity. Replica routing: can isolate a straggler, but risks inconsistent freshness or ACL visibility if replicas lag. Validate consistency and capacity. Lower ef, top k or aggressive quantization: can improve speed at the expense of retrieval recall and downstream answer quality. Timeouts and circuit breakers: prevent unlimited resource use; they may reduce successful answer rate. Measure errors and abstentions alongside latency. Percentile caveats: if a sample contains fewer than ~100 observations, p99 is unstable; histogram bucket resolution, sampling bias, coordinated omission and mixed query classes can also distort the metric. 7. Prevent recurrence Instrument distributed spans, request histograms and per peer/shard CPU, memory pressure, disk latency/IOPS, optimizations, queue waits, cache effectiveness, retries and request class dimensions. Qdrant exposes node and collection metrics via /metrics; scrape every peer, since node endpoints are not cluster wide. Define alert thresholds for p99 and p99/p50 ratio together with request volume , error rate, backlog growth and retrieval quality. Keep query class, tenant and region breakdowns where cardinality and privacy budgets allow. Run continuous workload benchmarks under simultaneous reads/writes, cold and warm caches, burst traffic and expected metadata filters. Load test new indices/configuration before promotion. Add guards for ingestion concurrency, query complexity and expensive filters; maintain on call playbooks for diagnosing CPU, disk, network, shard and query planner tails. After every incident, preserve an evidence backed timeline and verify the fix against p99, freshness, cost and recall , not median latency alone. Interview takeaway: When p99 explodes but p50 stays flat , isolate the slow request cohort and locate time in queue → coordinator → shard/filter → ANN → disk . Use traces, provider profiling and a reversible intervention to prove the bottleneck before increasing hardware or sacrificing recall. References Qdrant, Monitoring & Telemetry: https://qdrant.tech/documentation/ops monitoring/monitoring/ Qdrant, Database Optimization troubleshooting: https://qdrant.tech/documentation/faq/database optimization/ Qdrant, Indexing and storage choices: https://qdrant.tech/documentation/manage data/indexing/ Weaviate, Query profiling: https://docs.weaviate.io/weaviate/search/query profile

A vector database node reaches 95% memory usage during peak traffic. What would you investigate before adding more nodes?

Production incident (illustrative): A multi tenant RAG platform runs vector search on a 32 GiB node. During peak traffic, the dashboard reports 95% memory utilization and an on call engineer proposes doubling the node count. At the same time, some queries slow down, ingestion is active, and a replica restarts once. But “95% memory” alone cannot distinguish useful file system cache from non reclaimable memory, nor prove that adding nodes will solve the incident. Goal: Identify the actual memory consumer, prove whether there is harmful memory pressure, relieve the bottleneck safely, and scale only if measured steady state demand and required headroom justify it. 1. Contain the production incident Verify whether the node is in immediate danger: OOMKilled events, container memory limit breaches, failed allocations, increasing major page faults, sustained swap in/out, memory pressure stalls, elevated p95/p99, request errors and long GC pauses. If OOMs or errors are active, temporarily cap ingestion/optimizer concurrency and expensive concurrent searches, and route load to healthy, sufficiently provisioned replicas where available. Avoid removing security filters or disabling durability to save memory. Preserve dashboards, process/container metrics, deployment/config changes, collection/shard inventory and a representative failing search trace before restarting anything. Separate an alert on utilization from an incident of memory exhaustion ; high cache usage can be desirable and reclaimable. 2. Diagnose what “95% used” measures Read host MemTotal, MemAvailable, file/page cache, anonymous memory, slab and swap; compare with container/cgroup memory.current, memory.max, memory.stat (notably anon and file), memory.events OOM counters, and process RSS/PSS. Interpret values carefully: RSS includes memory mapped file pages, shared mappings can be counted in multiple processes, and host “used” memory may largely be reclaimable page cache. Container limits and truly available headroom matter more than a single dashboard percentage. Check time series of anonymous/resident memory, cache hit/miss ratio, memory pressure stalls and major faults. Is memory stable after warm up, growing without bound, or peaking only during index optimization? Attribute consumption by collection, shard, replica and component: raw dense/sparse vectors, HNSW graph, quantized copies, payload/filter indexes, ID mappings, mmap/page cache, loaded/growing segments, query buffers, connection pools, caches and ingestion/compaction scratch space. Inspect vectors per node, dimensions and dtype; replication/shard layout; large filters and oversized result payloads; pending optimizers/compactions; background batch concurrency; and active query concurrency. Don't assume every vendor stores these structures identically: Qdrant supports per structure memory tiers, Weaviate documents HNSW/index and vector cache requirements, and Milvus has query and growing/loaded segment behavior. 3. Trace cause rather than guessing Reclaimable cache: high reported host memory yet healthy MemAvailable, little reclaim/PSI pressure, no OOM and stable p99. The fix may simply be correcting alert thresholds and tracking meaningful signals, not scaling. Pinned vector/index/payload memory: anonymous memory and OOM risk rise with collection size; estimate vectors, graph links and payload indexes. Review per component memory tier and configured index dimensions before scaling. Temporary optimizer peak: memory increases in sync with compaction, segment merging, HNSW builds or bulk upserts. Compare active optimizers, work queues and peak memory against normal serving baseline. Request concurrency spike: query buffers and result allocations scale with simultaneous queries, large top k, nested filters, oversize payload fetches or co located workloads. Compare allocations per request and in flight concurrency. Skew or uneven placement: only one node/tenant/shard spikes while peers have space. Check shard sizes, local hot collections and replica routing before buying more capacity. Leak or cache misconfiguration: monotonically growing resident/heap memory despite stable dataset and throughput, never reclaimed after load subsides. Profile allocations and inspect deployments for regressions. I/O bottleneck masquerading as memory: moving vectors/index to disk relieves pinned RAM, but page cache misses and slow SSD/network storage can worsen latency. Investigate disk read IOPS/await and tail latency together. 4. Practical numerical example The node has 32 GiB of RAM and reports 30.4 GiB used = 95% . Breakdown from an illustrative host/cgroup investigation: 13 GiB non reclaimable anonymous/pinned process memory , 12 GiB mostly reclaimable page cache , 5.4 GiB other committed memory , and 1.6 GiB nominally unused . These categories total 32 GiB, while the first three total 30.4 GiB used; actual MemAvailable may be higher than nominally unused RAM because cache is reclaimable. Real kernel accounting can overlap: verify cgroup stats and MemAvailable rather than relying on this simplified inventory. The vector collection holds 2,000,000 vectors of 768 dimensions in float32 . Raw storage estimate = 2,000,000 × 768 × 4 bytes = 6,144,000,000 bytes ≈ 5.72 GiB , before HNSW links, payload indexes, ID tracking, quantized replicas, allocator overhead and temporary builders. During bulk ingestion, optimizer/build workers add an observed 5 GiB temporary allocation : illustrative non reclaimable demand rises from 13 to 18 GiB . A parallel batch of large filter searches adds 4 GiB , producing 22 GiB non reclaimable demand . On a node also running OS services and page cache, this triggers reclaim and latency spikes. Controlled test: cap ingestion builders from 4 to 1 , limit concurrent expensive searches from 40 to 20 , and keep the same validation workload. Peak non reclaimable memory falls from 22 to 16 GiB and vector search p99 from 850 to 290 ms , while the index remains complete. Those results support transient write/query concurrency as the proximate cause rather than permanent vector footprint growth. If demand instead settles at 28 GiB non reclaimable with an operational 25% headroom policy, required RAM is 28 / 0.75 ≈ 37.3 GiB , so a 32 GiB node is undersized even before replicas and build peaks. That is measured evidence to resize or rebalance. Headroom percentages are business specific, not universal. Compression illustration: a raw float32 vector uses 3,072 bytes at 768 dimensions; an int8 representation uses 768 bytes ( 75% less for that representation ), but actual node savings will be much smaller if graphs, raw vectors, payloads, caches or indexing buffers remain. Benchmark recall and rescoring before enabling quantization. 5. Apply fixes in risk aware order First: observation and safety. Fix misleading utilization dashboards; monitor OOM counters, MemAvailable, cgroup usage/limit, PSI, faults, swap, disk latency and search tail latency. Reduce unnecessary work only with clear freshness/latency limits. Second: reversible workload controls. Rate limit ingestion, cap concurrent optimizer tasks, smooth bursty traffic, reduce excessive query concurrency, prevent unbounded result sets and tune request caching. Verify no SLA regression and no failed writes. Third: correct skew and lifecycle waste. Rebalance existing shards/replicas, remove proven orphaned/superseded data through governed deletion, and review over indexed payload fields or duplicate vectors. Never delete active or legally retained data merely to free RAM. Fourth: tune representation and storage. Move appropriate structures to cached/cold/on disk tiers when supported; choose fast SSD/NVMe; quantize or use compression where validated. Index/payload changes may require rebuilds or optimizer passes, extra temporary capacity, and controlled rollout. Recheck Recall@k , filter correctness and p99, not memory alone. Fifth: scale with evidence. Add memory, rebalance, or add nodes when active non reclaimable working set plus build headroom exceeds safe capacity, when load is horizontally partitionable, and when projected growth justifies it. Extra nodes won't automatically fix a leak, poor placement, excessive per query buffers or a bad filter. 6. Trade offs and edge cases Tighter concurrency: protects memory/p99 but may increase queueing and source to search freshness lag; define separate ingestion and search service budgets. On disk vectors or HNSW: reduces pinned RAM but raises disk dependencies, page faults and potentially p99; mmap pages still appear as process memory. Memory tier options depend on vendor/version. Quantization: reduces stored representation size but may lose nearest neighbor recall, especially for borderline vectors. Original vector rescoring can recover quality at added I/O/CPU cost. Lower graph connectivity / smaller cache: reduces memory but can hurt Recall@k or slow search; evaluate real query/filter workloads before rebuild. Scaling out: may help when sharding distributes memory and QPS, but replicas can duplicate memory footprint and certain hot shards remain problematic. Verify placement and replication semantics. Aggressive garbage collection or restarts: can temporarily reduce apparent usage but may cause latency spikes, cold cache effects or rebalancing; do not use as the root cause fix. Kubernetes nuances: cgroup memory counters, limits, oom kill, Kubernetes working set approximations and host MemAvailable have different semantics. Use several corroborating signals rather than treating any one metric as exact available RAM. 7. Prevent recurrence Record a component level memory budget: vector data, HNSW graph, payload/filter indexes, replicas, per query buffers, optimizers, cache and deliberate headroom. Reforecast with vector counts, dimensions and data growth. Dashboard non reclaimable vs reclaimable memory, OOM events, swap/PSI, major faults, page cache hit rate, disk pressure, concurrent queries, background optimizations and p50/p99 per node and tenant. Alert on sustained pressure and projected exhaustion , not only 95% utilization. Correlate memory warnings with user facing latency/errors and ingest lag. Run representative peak traffic + bulk indexing load tests, cold/warm cache tests and growth projections. Include node restarts, failover, compaction and large filtered queries. Review configuration/rollout history and maintain a playbook for memory attribution, reversible mitigations, quality checks and capacity changes. Interview takeaway: 95% memory usage is a symptom, not a scale out decision. Determine whether memory is reclaimable cache, active vector/index footprint, temporary optimizer/query buffers, a leak, or shard imbalance. Fix the demonstrated cause and preserve search recall and freshness; scale only when the steady state plus peak headroom still exceeds safe capacity. References Qdrant: Database optimization FAQ — https://qdrant.tech/documentation/faq/database optimization/ Qdrant: Memory tiers — https://qdrant.tech/documentation/ops configuration/memory tiers/ Qdrant: Optimize performance — https://qdrant.tech/documentation/ops optimization/optimize/ Weaviate: Resource planning — https://docs.weaviate.io/weaviate/concepts/resources Milvus: Data processing and loaded segments — https://milvus.io/docs/data processing.md

Your HNSW index performs well initially but loses recall after millions of insertions and deletions. How would you diagnose index degradation?

Production incident (illustrative): An enterprise RAG retrieval service contains about 10 million active embeddings . Over six weeks, continuous document replacements cause millions of upserts and deletions. The service still answers queries quickly, yet its measured Recall@10 falls from 0.95 to 0.76 . Users report missing policy paragraphs that are confirmed present in source storage. An engineer proposes increasing ef search or rebuilding every index immediately. Both could help in some circumstances, but neither is a diagnosis. Key principle: HNSW is an approximate nearest neighbor graph. A recall drop after heavy updates does not prove HNSW graph corruption . Separate true graph/index degradation from missing or not yet visible vectors, tombstone/optimizer backlog, post filtering, stale replicas, embedding model drift, and benchmark errors. 1. Contain and reproduce the incident Confirm the severity: is this retrieval only Recall@k or end to end answer quality degradation? Monitor missing citations, zero/short result sets and query latency alongside recall. Stop destructive maintenance or aggressive configuration experiments until an index snapshot, recent deployment history, graph/index settings and live collection statistics are captured. If there is a reliable earlier collection/replica, consider a controlled read switch after verifying freshness and ACLs. Take a fixed, versioned evaluation set of real queries, expected relevant chunk IDs, tenant and filters. Ensure those expected chunks are still logically active, permitted and searchable. Do not count genuinely deleted records as false negatives. Construct an exact nearest neighbor baseline over the same active corpus with the same distance metric, embedding dimension, vector normalization and authorization/filter predicates (using flat/brute force scan or an exact reference path). Compare approximate top k against exact top k, and also measure relevance judgment recall separately. Exact neighbor recall and relevance recall are different metrics. Test on the same collection snapshot , same embedding model, same filters, query class, shard, replica and ef search, before and after each experiment. A comparison between differing corpora can manufacture an apparent regression. 2. Debug the data path before blaming the graph Trace source document → immutable chunk ID → embedding/model version → upsert acknowledgment → index/search visibility → retrieved ID. Verify expected live vector count, duplicate IDs, recent inserts, replay failures, source deletions and ACL/tenant filters. Query a few known missing chunk IDs directly by ID: Does the vector still exist? Is it active? Which shard/replica owns it? Does an exact search recover it under the original filters? Inspect optimizer backlog, deferred/unindexed points, tombstone/deleted point fraction, segment counts, compaction/merge history, HNSW build failures, replica divergence and index health per shard—not only cluster aggregates. Compare ingestion rate with index maintenance capacity: large non indexed segments, delayed optimization or stale replicas can make apparent recall inconsistent. Read visibility and optimizer semantics vary by vendor and configuration. Review workload changes: higher filter selectivity, larger top k, lower ef search, lower m or ef construction on rebuilt indexes, new quantization/rescoring settings, distance metric changes, dimension/model changes, or recent software versions. Root cause decision tree: If expected IDs are absent from authoritative storage, fix ingestion/lineage. If exact search has them but ANN misses them, investigate ANN configuration and graph/index health. If ANN works unfiltered but fails after filtering, examine filter strategy, candidate budgets and dead tuples. If only certain replicas fail, inspect placement and replication. If corpus neighbor recall is stable but task relevance falls, examine embedding changes and evaluation drift. 3. Investigate HNSW update/deletion behavior specifically Tombstones/dead entries: Many systems mark deleted vectors before physically reclaiming them. In Qdrant the Vacuum Optimizer processes segments with accumulated deleted points; Weaviate has configurable HNSW tombstone cleanup. Inspect whether cleanup is running, blocked, or falling behind. Do not assume that a given vendor's internal HNSW graph necessarily disconnects after deletes. Dead tuples and candidate budget: pgvector notes that HNSW can yield fewer results than requested because of dead tuples or filtering. Higher hnsw.ef search and iterative scans can help, subject to latency and memory budgets. Diagnose the version and behavior rather than extrapolating a pgvector issue to Qdrant or Weaviate. Graph quality and churn: If the vendor exposes graph health signals, inspect entry point availability, live/deleted nodes, neighbor counts, disconnected reachability, maintenance errors and index build configuration; otherwise rely on exact vs ANN recall under controlled snapshots. HNSW internals and exposed diagnostics are implementation dependent. Segment/index readiness: In Qdrant, heavy updates can leave large unoptimized segments; certain modes can defer search visibility of newly written vectors. Check indexed point and deferred point counters, optimizer thresholds and per segment status. An indexed only optimization can hide recent points, so do not enable it blindly in a freshness critical RAG system. Filter interaction: Compare identical unfiltered and filtered queries, especially highly selective tenant/ACL or policy version constraints. Never disable authorization filters in production to increase apparent recall. Version specific maintenance bugs: Investigate known issues and fixed releases when appropriate. For example, pgvector's 2026 releases included fixes related to HNSW vacuum/index repair. Confirm your actual installed version and maintenance status before attributing degradation to a known defect. 4. Numerical example: isolate the responsible mechanism Fixed evaluation set: 1,000 queries , each with a top 10 exact neighbor reference on the same active corpus : 10,000 exact reference neighbors in total, assuming ten distinct valid neighbors per query. Baseline: ANN returns 9,500 of 10,000 reference neighbors → Recall@10 = 95% . After sustained churn it returns 7,600 of 10,000 → 76% , a 19 percentage point drop (20% relative to the baseline). On the affected shard, suppose there are 3 million live points plus 1 million marked deleted points . Deletion fraction = 1 / (3 + 1) = 25% , which exceeds an illustrative Qdrant vacuum threshold of 20% ; this alone is evidence to inspect optimizer operation , not proof it caused lost neighbors. The investigation finds two independent issues: 800 expected active neighbors exist in source storage but are absent from the query visible index due to optimizer/visibility backlog; the other 1,100 are visible through exact search but absent from ANN top 10 because of the current search candidate/filtered query behavior. These counts partition the observed 1,900 missing neighbor instances in this illustration; repeated IDs across queries are counted per evaluation occurrence. Reconcile/replay the missing active points and wait for confirmed visibility, then measure. Suppose Recall@10 rises from 76% to 84% . Raise a test only search budget and/or enable vendor supported iterative filtered search on the same stable snapshot; suppose it reaches 93% , with p99 rising from 150 ms to 240 ms . A controlled healthy shadow index built from active records achieves 95% at 175 ms , suggesting the maintenance/index configuration deserves deeper attention. Choose the intervention based on evidence : repair ingestion/index readiness first; if persisted ANN quality remains poor with fixed parameters and adequate candidate budgets, rebuild the affected index/segment into a shadow copy and compare before switching serving traffic. 5. Engineering recovery plan Restore logical correctness: Reconcile source of truth active IDs against durable vector store inventory, replay missed events idempotently, and confirm every recovered point is query visible before re running recall evaluation. Recover maintenance health: Inspect and tune the vendor's optimizer/vacuum/tombstone cleanup thresholds and resource budgets. Allow maintenance to catch up without saturating live search. Verify its completion with counters—not simply by restarting a worker. Tune retrieval carefully: Benchmark ef search, filtered query planning, iterative scans and rescoring on representative queries; preserve ACL filters and record Recall@k, p95/p99, CPU, memory, and freshness. Rebuild only when justified: If exact vs ANN comparison still implicates graph/index quality, perform a versioned shadow rebuild from an immutable active corpus snapshot, incorporate a bounded change log for writes during rebuild, verify counts/hash/ACLs and recall, and cut over using a supported atomic alias or routing pointer method. The exact cutover semantics depend on the vendor; never assume a multi service global transaction. Rollback: Keep the last verified serving index temporarily, compare canary recall and latency, and revert the routing pointer if degradation persists. Ensure recent deletions and access revocations remain enforced even during rollback. 6. Trade offs and common mistakes Higher search effort: may improve ANN recall but increases CPU, memory and query latency. It cannot recover a vector that was never ingested or is excluded by an eligibility filter. Aggressive vacuum/compaction: reduces dead records and backlog, but competes for CPU, I/O and temporary storage; scheduling and concurrency affect live p99. Frequent full rebuilds: can restore graph/index structure and consolidate deletes, but cost compute, temporary disk, write catch up, and operational risk. Avoid using rebuilds as a substitute for finding missing points. Lower optimizer thresholds: may trigger more frequent work; higher thresholds preserve more tombstones and can delay cleanup. Tune empirically for actual update patterns. Changed embeddings or distance metric: an old golden test set or mixed embedding spaces can make relevance/neighbor comparisons invalid; version query and document embeddings together. Filtered queries: post filtering may yield fewer than k candidates; naive removal of security filters is unacceptable. Evaluate exact and ANN with identical tenant/ACL/effective version predicates. Evaluation bias: recall estimates based on a small sample, changing ground truth, stale replicas or inconsistent snapshots can falsely indicate decay. Publish query set version, eligible corpus snapshot, filter definitions and recall confidence intervals. Mistaking latency for recall: a search that returns quickly can still be wrong; a slower exact scan may reveal missing candidates but is not automatically a production serving strategy. 7. Prevent recurrence Maintain per shard live/deleted/deferred/visible vector counts, optimizer backlog and duration, graph/index build errors, replica lag, source to vector reconciliation results and fresh insert visibility lag. Run a scheduled exact vs ANN Recall@k canary on versioned query samples and stable source snapshots, broken down by tenant, filtered/unfiltered traffic, data age and shard. Alert on meaningful drops with minimum sample sizes. Enforce idempotent source version events, deterministic chunk IDs, transactional publication manifests, deletion tombstones and authorization predicates before serving. Forecast churn (% inserts, updates and deletes), provision optimizer CPU/I/O and memory headroom, and keep a periodic maintenance window with observable success criteria. Regression test heavy insertion/deletion sequences (including concurrent deletes, reinserted IDs, selective filters and failed optimizers) against the actual vendor/version. Keep a tested shadow rebuild/canary/rollback runbook, but invoke it only when validation and maintenance remediation fail to restore expected quality. Interview takeaway: When HNSW Recall@k falls after millions of writes, first establish an exact baseline over the same live corpus , then separate missing/index invisible vectors, tombstone or optimizer lag, and ANN graph/search budget effects . Prove the cause on a reproducible snapshot before increasing ef search or rebuilding the index. References Qdrant, Optimizer and Vacuum Optimizer: https://qdrant.tech/documentation/ops optimization/optimizer/ Qdrant, Predictable search latency and optimizer tuning: https://qdrant.tech/documentation/production operations/predictable search latency/ Weaviate, Vector index and tombstone cleanup controls: https://docs.weaviate.io/weaviate/config refs/indexing/vector index pgvector, Index searches / iterative scans / HNSW filtering and dead tuples: https://github.com/pgvector/pgvector pgvector, HNSW maintenance changelog: https://github.com/pgvector/pgvector/blob/master/CHANGELOG.md

A large index rebuild takes six hours. How would you keep search available during the rebuild?

Production incident (illustrative): A 12 million vector RAG service needs to rebuild its HNSW index after an index format change. A straightforward rebuild takes six hours . The first attempt rebuilds the live collection in place: memory and disk I/O spike, retrieval p99 rises from 180 ms to 2,000 ms , and some queries time out. Stopping search for six hours is unacceptable. The problem is the deployment strategy , not necessarily a faulty ANN algorithm. 1. Contain and identify the bottleneck Keep the last verified index serving queries through a stable knowledge prod alias or application routing pointer. If an in place rebuild is already damaging service, stop or throttle it when the backend permits safely; restore traffic to a healthy replica or pre rebuild collection if available. Capture the rebuild stage and worker logs, read/write IOPS, disk capacity, RAM/page faults, optimizer CPU use, query queues, shard health, p50/p95/p99, errors, recall, ingest lag and current ACL/deletion state. Identify whether failure is caused by read/write contention, the unavailability of the index being rebuilt, optimizer pauses, poor resource isolation or premature cutover before all writes are visible. Avoid assuming replicas solve everything: if all replicas share the same collection update or disk bottleneck, they can degrade together. Confirm isolation and vendor semantics. 2. Use blue/green rebuilding with isolation Blue: rag index v1, currently serving all reads through knowledge prod. Green: rag index v2, a separately named collection/index with matching tenant, metadata filter, authorization and citation schemas. Create it with enough independent CPU, RAM, I/O and storage to avoid starving blue. Freeze a consistent source snapshot at durable change log offset L0. Include immutable chunk IDs, document/version manifest, source hashes, embedding model version and an ACL/deletion ledger. Build green from the authoritative snapshot without changing blue. Continue writes to blue and record every change since L0 in an ordered durable change log (CDC/outbox): inserts, upserts, updates, deletes, permission revocations and document supersessions. If carefully implemented, optional dual writing can reduce catch up lag, but is not a substitute for an independently replayable log and idempotent reconciliation. Build green in the background, then replay log entries from L0 in per document/version sequence . Use deterministic IDs and idempotent operations. Fence against late older writes, and apply deletions/ACL revocations to both indices or at a shared serving gate immediately to prevent re exposure. When green is near real time, briefly fence/synchronize the write stream: capture cutover offset L1, ensure green is caught up and query visible through L1, then switch the alias or routing pointer. Continue event delivery to the new primary; do not drop writes during the switch. In Qdrant, an atomic multi action alias update can redirect a stable alias to the new collection. Weaviate's collection alias update is also atomic. Confirm the specific provider/version behavior, especially for in flight requests, replicas and writes to an alias. Keep blue retained for a bounded rollback window. A rollback is safe only if deletes, revocations and writes after cutover have been propagated to blue or are enforced independently; an outdated blue index is not a safe rollback target . 3. Validate before cutover Completeness: reconcile snapshot manifests with green's live IDs, counts and document versions; use hashes and sample individual records—not counts alone. Confirm that active records are present and deleted records are absent/blocked. Consistency: verify CDC cursor and end to end read visibility through L1, query time freshness, current ACL/tenant filters and citation targets. Protect against async replica lag and stale caches. Search quality: run golden queries on both versions, compare Recall@k against an appropriate exact/labelled reference, nDCG, answer citations, filter correctness and p95/p99. If the rebuild also changes embedding model or metric, evaluate each against its own valid reference and compare task relevance on labelled queries; don't conflate vector spaces. Operational health: ensure no failed shards, no growing ingestion backlog, sufficient resource headroom and p99 below agreed SLO under concurrent writes. Release: canary a small cohort to green using an application level router if the backend alias only supports a whole collection switch. After passing tests, perform one guarded alias cutover. Monitor errors, quality and source to search freshness; invalidate caches keyed to the old index version. 4. Numerical example: six hour rebuild without downtime Blue has 12,000,000 active vectors . At 10:00 , capture snapshot and start green build at log offset L0 = 50,000,000. Blue continues serving queries. Average change rate is 100 events per minute for 360 minutes : 36,000 inserts, updates, deletions and ACL/version operations accumulate during the six hour build. Green finishes at 16:00 with the snapshot's 12,000,000 records. Assume a dedicated replay worker applies 600 events/minute while fresh changes continue at 100/minute. Net backlog reduction = 600 − 100 = 500 events/minute . First order catch up time is 36,000/500 = 72 minutes , ending around 17:12 . This assumes steady rates, no bursts and no replay failures. At 17:12 , freeze a short cutover checkpoint L1, wait for green replay and search visibility, then switch knowledge prod: v1 → v2. Suppose a coordinated fence lasts 15 seconds : query traffic remains on blue during the fence, so read downtime is 0 seconds ; write acceptance may queue briefly, and this must be tracked separately from read availability. Sample validation: 12,000,000/12,000,000 snapshot records accounted for by the manifest, every event through L1 reconciled, 1,000/1,000 security filter canary queries pass, and representative Recall@10 remains ≥0.94 (illustrative acceptance criterion). p99 stays ≤200 ms during the rebuild because the green resource budget doesn't starve blue. The naïve in place rebuild would risk 360 minutes of degraded or interrupted service. The isolated rebuild serves blue continuously for 432 minutes of build plus catch up, then changes a pointer; this improves availability , not rebuild duration. If green cannot pass gates, continue blue service rather than forcing a cutover. 5. Engineering choices and trade offs Separate collection / blue green: simple rollback and no read interruption, but temporary extra vectors, storage, network and compute. Peak disk may exceed 2× for segments, WAL, compaction and snapshots; capacity plan with headroom. Replica by replica rolling rebuild: useful where the backend supports independent replica rebuild and quorum compatible reads/writes, but can reduce redundancy or cause version divergence. Prove consistency rather than assuming it. Dual writes: reduce catch up lag but introduce partial success, ordering races and drift; durable CDC plus reconciliation is usually easier to audit. Short write fence: simplifies an atomic cutover but increases write latency/backlog. For strict continuous writes, use an ordered router/outbox and generation fencing so accepted writes are never lost or applied to only the retired collection. Hot swap alias: fast control plane operation, but not an atomic transaction over independent blob stores, caches, CDC and policy ledgers. Verify those separately. Keeping blue for rollback: costs space and creates risk from stale ACLs/deletes; use a shared deny serving tombstone gate plus replay of newer events. Resource isolation: speeds/stabilizes production queries but can increase infrastructure cost. Without isolation the rebuilt green index can consume all disk I/O and degrade blue even without a serving cutover. Freshness vs stability: delaying cutover for full reconciliation protects correctness, but extends how long the old search configuration is served. If blue no longer meets safety/quality gates, use an approved degraded retrieval mode or controlled fail closed response. 6. Prevent repeat incidents Make versioned collections, stable aliases, immutable snapshots, durable CDC replay and guarded promotion part of the standard production pipeline. Maintain read p99/error and retrieval quality SLOs alongside write acceptance and source to index freshness SLOs. Rehearse failures: builder crash at 95%, replay duplicates/out of order deletes, change log retention expiration, alias update failure, ACL revocation immediately before cutover, stale replica, failed rollback and full disk. Monitor build progress, CPU/I/O impact on blue, event backlog, per shard record/visibility counts, CDC offset lag, delete propagation, filter correctness and green canary quality. Use a sign off checklist: manifest reconciled, event log replayed through a committed checkpoint, no security regressions, recall/latency validated, rollback index safe, and operator/audit evidence captured. Interview takeaway: Do not rebuild the only serving index in place. Keep blue serving; build green from a snapshot, replay every intervening mutation, verify query visible completeness and permissions, then atomically switch the read alias. A zero downtime pointer switch is not enough unless writes, deletes, ACLs and caches are also consistent. References Qdrant, Collection aliases (atomic switch): https://qdrant.tech/documentation/manage data/collections/ Weaviate, Collection aliases: https://docs.weaviate.io/weaviate/manage collections/collection aliases Weaviate, Zero downtime collection migration: https://docs.weaviate.io/weaviate/tutorials/collection aliases Elastic, Alias operations and reindex without downtime: https://www.elastic.co/docs/manage data/data store/aliases

One vector-search shard consistently responds much slower than the others. How would you identify and mitigate the hot shard?

Production incident (illustrative): A customer support RAG system searches a 12 million vector collection across six shards . At peak load, the search coordinator waits nearly a second for shard 4 while the other shards return in roughly 60–90 ms. Search p99 rises, and requests time out even though the cluster wide average CPU looks normal. A proposal to add two nodes misses the central question: why is one shard hot? Core principle: A hot shard can result from disproportionate query or write traffic, larger indexed data, expensive metadata filters, a slow host, or a lagging/compacting replica . Identify whether the problem follows a logical shard , a physical node , or a query class before moving data. 1. Contain the production incident Quantify affected search requests, p95/p99, timeout and error rates. Preserve representative traces and the placement map: collection → shard id → replica id → node id. Keep tenant and ACL filters enabled. If safe, slow down background ingestion/optimizers, bound expensive query concurrency, or route reads to healthy, sufficiently current replicas that meet the required consistency and authorization guarantees. Avoid immediately resharding or restarting the slow node. Both can increase disk I/O, trigger cold cache penalties, or hide the original cause. Check whether a single slow shard actually lies on the critical path. Fan out search often waits on the slowest participating shard, but execution and timeout semantics depend on the database. 2. Trace the bottleneck at shard and request level Instrument spans for client wait → coordinator planning → shard fan out → per replica queue → filter planning → ANN traversal → payload fetch → merge/rerank. Tag collection, shard, peer, request class, tenant (safely aggregated), top k, filter selectivity, index parameters, retries, queue time and source/version eligibility. Compare per shard QPS, read/write ops, active points, bytes/index footprint, HNSW and payload index sizes, deleted point ratio, optimizer progress, CPU throttling, RSS/page faults, memory pressure, disk read/write latency, cache misses and p50/p95/p99. Inspect data distribution: Did consistent hashing place a large tenant's records together? Does a user defined shard key route almost all requests to one shard? Does a time based scheme send nearly all recent data queries to today's shard? Cross check workload shape: A shard can contain the same number of vectors as its peers but receive many more queries, expensive predicates or high write/compaction volume. Examine replica placement and hardware: one slow replica may have degraded SSD I/O, CPU contention, poor network, stale index/cache or compaction backlog. Compare another replica of the same logical shard . Verify whether shard selection keys are present. In Qdrant user defined sharding, a query without the intended shard selector may search all shards; that broadens fan out and can overload the system. Do not route solely on untrusted tenant input. 3. Root cause decision tree Slow only on one physical peer: compare the same shard replica on other peers, hardware, cache, local indexing and maintenance; a targeted replica replacement/move may help. Slow logical shard across replicas: examine shard volume, dominant tenant/QPS, filter complexity, hot keys and compaction; moving its data unchanged may only move the hot spot. High queue wait, normal execution time: investigate burst concentration, connection pools, CPU scheduling and request routing. Cap or partition concurrency before raising hardware capacity. High ANN/filter execution time: inspect ef search, top k, selectivity, missing payload indexes and the query mix. Preserve retrieval recall and mandatory tenant/ACL filtering. High I/O and page fault time: measure hot index working set, memory tier, cache churn, SSD latency and co located ingestion. Test maintenance throttling or better I/O isolation. Write or optimizer hotspot: monitor update distribution and maintenance queue growth. Separate write heavy tenants or stagger ingestion without violating freshness budgets. Network/coordinator bottleneck: inspect retries, cross zone traffic and per shard fan out; shard movement won't cure a congested coordinator. Rule out misleading comparisons: use the same query mix, observation period, replica consistency setting and filters when contrasting shards. 4. Practical numerical example Six shards each hold 2 million vectors , giving 12 million total. Balanced storage does not guarantee balanced throughput. At peak, the cluster handles 1,200 searches/second . Shard 4 receives 600 shard targeted searches/second (50%) , while each of the other five receives 120 (10%) . The average is 200/shard/second ; shard 4's traffic is 3× the average and 5× each other shard . A fan out or targeted query trace shows shard 4 queue wait 650 ms + filter/ANN work 190 ms + payload fetch 60 ms = 900 ms . Another shard takes 15 + 55 + 10 = 80 ms . In requests requiring both shards, coordinator latency cannot beat the ~900 ms straggler, even before merge/network overhead. These are individual illustrative traces , not summed p99 percentiles. Logs reveal that one enterprise tenant uses a custom shard key and generates 480 of shard 4's 600 QPS , while other tenants contribute 120 QPS. Node CPU is high but raw vector counts are balanced. A controlled test partitions this tenant into four approved shard keys with stable document to key routing and a tested query fan out/merge plan. Assume its 480 QPS is evenly split across four eligible partitions (120 QPS each), leaving the original shard's remaining 120 QPS. This requires true data repartitioning ; adding read replicas alone increases read capacity but does not change shard key distribution. After migration, illustrative maximum shard targeted load falls from 600 to 240 QPS in the new balanced placement; slow shard p99 falls from 900 ms to 180 ms and overall search p99 from 1,050 ms to 260 ms . Validate the actual partition map and load—these improvements are an example, not a guaranteed scaling result. Compute improvement: overall p99 reduction = (1,050 − 260)/1,050 × 100 ≈ 75.2% . Compare Recall@10, ACL correctness, ingestion freshness and storage cost before approving the cutover. 5. Engineering mitigation options Bad replica/host: shift search to a verified healthy replica and repair or move the faulty replica. In Qdrant, inspect collection cluster info, peer placement and shard transfer status; transfer mechanisms can preserve or rebuild indexes differently. Hot tenant/key: partition a large tenant's logical shard, choose a better custom sharding strategy, or employ supported online resharding; document how reads combine results correctly across the newly assigned partitions. Retain versioned tenant to shard routing and audit ACLs. Oversized shard: rebalance/reshard with a snapshot and live change log catch up, checking that all vectors and deletions are accounted for. A move redistributes placement; a split redistributes logical work. They are not interchangeable. Filter heavy workload: index the relevant metadata fields, tune ANN/filter planning and measure exact vs approximate recall with the same filters; never drop access predicates. Burst induced queues: introduce per tenant quotas, bounded concurrency, fair scheduling, request coalescing where safe and workload isolation. Monitor whether throttling raises user facing 429s or freshness delays. Optimizer/I/O contention: rate limit maintenance/ingestion or provision independent fast storage, with a measured freshness versus p99 budget. Roll out via canary traffic, watch per shard QPS/queue/IO/latency and query correctness, and keep a secure rollback plan. On a physical move, verify target replication and read visibility before retiring the source. On logical repartition, retain consistent tenant routing plus old key reconciliation until fully migrated. 6. Trade offs and mistakes More replicas can increase read throughput and availability, but duplicate storage/RAM and may not fix hot key routing, write hotspots or query planning. More shards distributes load when the shard selection strategy permits it, but increases coordination, open segments, network fan out and operational complexity. Moving a shard can cure a bad node but merely relocates a logically hot shard. Moving heavy shards also consumes disk/network resources and can impact p99 during migration. Per tenant isolation improves fairness and predictable SLOs, but complicates onboarding, costs and cross tenant aggregations. Never mix tenant data or relax authorization. Caching can offload repetitive searches but may serve stale/deleted content unless keyed and invalidated by tenant, ACL and document version. Lower ANN search effort improves latency at potential Recall@k cost; compare quality before changing it. Coarse shard averages hide bad replicas and hot query classes. Diagnose per shard and per replica histograms, with enough requests for stable percentiles. 7. Prevent recurrence Dashboard shard and replica QPS, in flight/queued searches, p50/p95/p99, CPU and disk contention, index bytes, point count, cache hit rate, optimizer backlog, replica lag and dominant tenant/key shares. Alert on load skew (maximum shard QPS / mean shard QPS), queue age SLO breaches, per shard tail latency, missing shard selectors and rapidly growing shard sizes—not only average node CPU. Test tenant heavy and time skewed workloads with realistic top k, filters, simultaneous writes and cold caches; verify routed query results against an exact or trusted baseline. Establish sharding strategy and capacity planning rules that account for data distribution and request distribution ; forecast tenant and time window hotspots before production growth. Maintain rehearsed runbooks for replica isolation, safe shard move, logical split, resharding, write replay, canary/rollback and security validation. Interview takeaway: Hot shard diagnosis is about who sends work to which shard, where that shard is placed, and what consumes its time . Compare per shard/replica traces, queue pressure and load distribution first. Rebalance a faulty host, but split or reroute a truly hot logical partition rather than simply moving its overload to another node. References Qdrant, Distributed deployment: https://qdrant.tech/documentation/scaling/distributed deployment/ Qdrant, Monitoring: https://qdrant.tech/documentation/ops monitoring/monitoring/ Qdrant, Multitenancy: https://qdrant.tech/documentation/guides/optimize/ Weaviate, Multi tenancy operations: https://docs.weaviate.io/weaviate/manage collections/multi tenancy