Your ingestion workers process the same message multiple times. How would you prevent duplicate vectors and inconsistent document versions?

Production incident: A RAG ingestion queue delivers the same “index document” event to two workers. Both generate embeddings and write vectors. The search index now contains duplicate chunks, or an older document version overwrites a newer version. This is usually a message-processing idempotency and concurrency problem, not simply a duplicate-file problem.

Why can the same event execute more than once?

• At-least-once delivery: Many queues intentionally redeliver messages if acknowledgments are lost or delayed.
• Visibility timeout / lease expiry: Worker A is still embedding when its lease expires; Worker B receives the same message.
• Crash between write and acknowledgment: The vector upsert succeeds, but the worker dies before acknowledging the message.
• Producer retries: A producer submits the same logical update more than once, sometimes with different queue message IDs.
• Out-of-order versions: An event for version 7 finishes after an event for version 8.

AWS documents at-least-once SQS delivery and redelivery when visibility timeouts expire; a queue-level deduplication feature does not replace safe application writes.

Step-by-step root-cause investigation

1. Pick one affected tenant_id/document_id and inspect the source version, queue message IDs, delivery attempts, worker IDs, processing start/end times, and acknowledgment records.
2. Query the vector index by document ID and version. Compare its chunk IDs/count with the expected chunk manifest.
3. Check whether write operations used random vector IDs (duplicate inserts) or deterministic IDs (safe updates for identical content).
4. Search worker traces for lease expiration, retry storms, lost acknowledgments, or two workers handling the same document concurrently.
5. Compare the persisted active version with message versions to detect stale workers overwriting newer data.
6. Reproduce under controlled fault injection: pause a worker after its vector write but before acknowledgment, then redeliver the event.

Engineering solution: design for repeated delivery

1. Stable operation identity: Derive an idempotency key from tenant_id + document_id + source_version + pipeline_version. Use an authoritative version or content hash; do not rely only on the queue message ID.
2. Claim work atomically: In a durable job ledger, create or claim a unique processing record. Mark duplicates as already processing/completed when appropriate. Expired claims must be recoverable.
3. Deterministic vector IDs: Use a stable chunk identity, for example hash(tenant_id, document_id, source_version, chunk_index, chunker_version). Write with upsert rather than insert-with-new-random-ID.
4. Fence stale versions: Track latest_accepted_source_version and an active index generation. Before activating a version, atomically verify that it is still current. A worker finishing version 7 must not make it active after version 8 has won. A lock alone is insufficient if an expired worker can continue writing; use a monotonic fencing token or transactional compare-and-swap.
5. Publish atomically at the logical level: Stage a document version's chunks under a generation/version; switch the active pointer only after expected chunks are present and searchable. Queries must filter to the active version. This avoids partially updated results even if vector writes are not transactional.
6. Acknowledge only after durable completion: Commit the document state after the required downstream writes/verification. Then acknowledge the queue message. A retry following an acknowledgment failure should detect the completed operation and acknowledge without reindexing.
7. Safe cleanup: Reconcile abandoned generations and delete superseded chunk IDs only after the new active version is safely available.
8. Recovery: Apply bounded retry/backoff and dead-letter handling. Investigate poison messages instead of retrying indefinitely.

Numerical example

Suppose 1,000 unique document-update events are delivered 1,150 times due to retries and lease expiry. Each update produces 20 chunks.

• With random IDs on every attempt: 1,150 × 20 = 23,000 vector rows may be inserted.
• With one committed generation per logical event: 1,000 × 20 = 20,000 intended vector rows.
• Potential excess: 3,000 rows, or 15% more rows than intended.

These values are illustrative and assume each event represents a distinct accepted version and has 20 chunks. In a real system, several versions of one document may exist temporarily during a safe cutover.

Minimal processing sketch

event = (tenant, document_id, source_version, pipeline_version)
key = stable_hash(event)

if ledger.is_completed(key):
    acknowledge(event)
    return

claim = ledger.try_claim(key)        # atomic, lease/fencing protected
if not claim:
    retry_later(event)
    return

chunks = parse_and_chunk(event)
upsert_vectors(deterministic_ids(chunks, event), generation=key)

if verify_expected_chunks(key) and
   ledger.activate_if_current(document_id, source_version, claim.fence):
    ledger.mark_completed(key)
    acknowledge(event)
else:
    mark_for_reconciliation(event)

Treat this as pseudocode, not copy-paste production code: actual implementations must handle lease renewal, transactions, and failures between operations.

Trade-offs and prevention

• Stable IDs + upserts reduce duplicate writes, but do not alone prevent old versions from replacing new ones.
• Per-document serialization simplifies ordering but limits throughput for frequently updated documents; version fencing supports higher concurrency with additional complexity.
• Staging and active-version pointers add storage and bookkeeping but greatly reduce partial-index visibility.
• Queue exactly-once settings may reduce deliveries; they do not automatically make external vector/database side effects exactly once.

Monitor duplicate-delivery rate, idempotency-key conflicts, stale-version rejections, orphaned generations, expected-vs-actual chunk counts, redelivery attempts, lease expirations, and dead-letter counts. Add a load test that deliberately kills a worker after a successful vector write and before acknowledgment.

Interview takeaway: Assume messages can repeat. Combine business-level idempotency keys, deterministic vector upserts, durable work state, version fencing, and active-generation cutover so repeated or out-of-order processing never corrupts the searchable corpus.

References:

• AWS SQS — At-least-once delivery: https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/standard-queues-at-least-once-delivery.html
• AWS SQS — Visibility timeout: https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-visibility-timeout.html
• Google Cloud Pub/Sub — Exactly-once delivery and limitations: https://docs.cloud.google.com/pubsub/docs/exactly-once-delivery