Retrieval-Augmented Generation

Retrieval-augmented generation connects a generative model to a searchable knowledge source. Instead of relying only on model weights, the system fetches relevant documents and places them into the prompt or context.

How do you measure reranking latency in RAG?

Reranking latency in RAG is the elapsed time a reranking stage takes to score or reorder a retrieved candidate set before the best evidence reaches the language model. Instrument it separately from first stage retrieval and answer generation . Beginner friendly example A retriever returns 20 passages . A cross encoder evaluates the query against those passages, then returns the best 5 in a new order. Suppose five reranking requests take 40, 50, 60, 70, and 130 ms . Mean latency: (40 + 50 + 60 + 70 + 130) / 5 = 70 ms . Median latency (p50): 60 ms . Maximum: 130 ms . Five measurements are too few for reliable p95 or p99 estimates. Gather representative production traffic and report the percentile method. How do you measure it? 1. Set boundaries: Start a timer when the reranker receives the query and the candidate passages; stop after it returns relevance scores or an ordered list. Specify whether network transport, batching, tokenization, or queueing is included. 2. Trace the internals: For hosted models, separately track client observed request time and server processing when available. For local cross encoders, record preprocessing, batch preparation, inference, and sorting. 3. Record workload parameters: Candidate count, input token lengths, batch size, reranker model/version, CPU/GPU type, cache state, and concurrent requests. 4. Aggregate by segment: Report p50/p95/p99 latency, mean, error/timeout rates, and throughput separately for different candidate counts and model versions. 5. Measure usefulness, not only speed: Compare reranked nDCG/MRR or judged answer bearing evidence with added latency and cost. 6. Benchmark under load: Test realistic concurrency, cold starts, rate limits, and long documents, not just a single warmed up request. Numerical candidate count example Consider a simplified sequential reranker that takes 4 ms per query–passage pair : 10 candidates: about 40 ms of pair scoring time. 30 candidates: about 120 ms of pair scoring time. This assumes no batching or parallelism. Modern implementations can process several pairs together, so real latency need not scale linearly. Measure rather than extrapolate blindly. What should the trace look like? Common mistakes Including embedding or vector search time inside the reranking metric without saying so. Comparing 10 candidate runs with 100 candidate runs as if their workload were identical. Ignoring long passage lengths, GPU queueing, network retries, or cold starts. Optimizing milliseconds while losing useful evidence and worsening answer quality. Measuring only averages and hiding slow tail behavior. Interview takeaway: Reranking latency is a bounded query plus candidates to ranked results span . Track candidate count and input length, analyze tail latency, and evaluate its quality improvement against the added time and cost.

How do you measure end-to-end RAG latency?

End to end RAG latency is the elapsed time from when an application receives a user request until its answer is delivered. Unlike embedding, retrieval, or reranking latency, it includes all stages and coordination overhead . Beginner friendly example An assistant measures the following stages for one request: Query processing and embedding: 80 ms Vector/keyword retrieval: 120 ms Reranking: 70 ms Context construction: 30 ms LLM generation: 900 ms Network/application overhead: 100 ms Total end to end latency = 80 + 120 + 70 + 30 + 900 + 100 = 1,300 ms (1.3 seconds) . This addition assumes the stages run sequentially. Parallel branches overlap, so the actual duration must be measured using the request's wall clock start and finish times—not by summing every nested span. What should you measure? 1. Start and end boundaries: Start when the server receives the question; stop when the complete answer has been transmitted or the request has failed. State whether client network time is included. 2. Stage spans: Trace authentication/query preparation, embedding, retrieval, reranking, context assembly, generation, and response transport. 3. Streaming metrics: Record time to first token (TTFT) and time to complete response separately. A fast first token does not mean the whole answer was fast. 4. Distributions: Report p50, p95, and p99 over many real requests. Also monitor timeout and error rates; do not hide failed requests. 5. Request context: Record query type, retrieved candidate counts, input/output tokens, cache hits, model version, shard fan out, and tenant safe trace IDs. 6. Critical path: In concurrent retrieval, use span timing to identify the slowest required branch and serial dependencies. Numerical percentile example Suppose ten completed requests take (seconds): 0.7, 0.8, 0.9, 1.0, 1.0, 1.1, 1.2, 1.4, 1.8, 3.0 Mean: 12.9 / 10 = 1.29 s. Median: (1.0 + 1.1) / 2 = 1.05 s. Slowest: 3.0 s. This sample is too small for stable p95/p99 estimates. In production, use a documented percentile calculation method with sufficient observations. What commonly drives slow responses? Slow embedding APIs or cache misses. Large Top K retrieval, heavily filtered search, or slow shards. Expensive cross encoder/LLM rerankers. Oversized prompts and long generated answers. Model queueing, retries, rate limiting, and network overhead. How should you optimize? 1. Profile the largest contributors to the critical path , not just whichever stage appears most complex. 2. Improve retrieval selectivity, parallelize independent searches, and use caching where authorization and freshness permit. 3. Reduce irrelevant context and control output length without sacrificing answer correctness. 4. Re test groundedness, relevance, citations, and failure handling after every latency optimization. Interview takeaway: Measure user visible end to end latency with one wall clock trace , split it into stage level spans, and track TTFT, full completion, tail percentiles, and answer quality .

How do you monitor RAG systems in production?

Monitoring a production RAG system means continuously checking that ingestion, retrieval, model generation, and downstream services are functioning—and that users are receiving timely, authorized, evidence grounded responses. Monitoring is an operational process , not just a list of quality metrics. Beginner friendly example A company handbook assistant works well on Monday. On Tuesday, a document ingestion job fails, so an updated leave policy never reaches the vector index. The model still produces fluent answers, but they rely on yesterday's rules. Production monitoring should detect failed ingestion and stale indexes , identify affected requests, alert the on call team, and confirm the repair before declaring the issue resolved. A practical monitoring workflow 1. Define service expectations: Establish service level indicators (SLIs) such as successful answers, retrieval availability, p95 response time, source freshness, and task specific quality. Set service level objectives (SLOs) against a baseline and business risk—not arbitrary universal thresholds. 2. Instrument the entire request: Give each request a privacy safe trace ID, then connect spans for authentication, query embedding, vector/keyword retrieval, reranking, context assembly, LLM generation, and response delivery. 3. Monitor ingestion separately: Track queued documents, parse failures, embedding errors, indexing lag, deletions, and the age of the latest searchable source version. 4. Create dashboards: Combine request volume, error rates, latency percentiles, token/cost use, cache hit rate, empty retrieval results, and sampled answer quality assessments. Segment by pipeline version, data source, and tenant where allowed. 5. Add quality checks: Review sampled answers for grounding, citation validity, relevance, and appropriate abstention. Automated judges must be validated against human review and should not be treated as flawless. 6. Alert on actionable failures: Page for outages, access control breaches, and sustained SLO violations. Use lower priority alerts or tickets for gradual freshness, cost, and quality regressions to avoid alert fatigue. 7. Investigate using traces: Follow a failing request through retriever, index, reranker, and model spans. Compare changes in source versions, model/config releases, and workload patterns. 8. Close the loop: Maintain incident runbooks, safe fallback behavior, rollback procedures, post incident reviews, and recurring regression tests. Numerical example Suppose a RAG service receives 2,000 requests in one hour and 40 fail. Request error rate: 40 / 2,000 × 100 = 2%. If its agreed error rate SLO is below 1% , this period breaches that target. Separately, if newly approved documents must be searchable within 15 minutes but indexing lag reaches 45 minutes , the freshness objective is breached even if request error rate is low. The targets are illustrative ; choose actual thresholds based on the application's requirements. What should happen when monitoring finds a problem? High error rate: Check dependency health, timeouts, quotas, and fallback paths. Slow responses: Inspect stage spans and the critical path before optimizing. Stale or missing knowledge: Check ingestion pipelines, source syncs, and index refreshes. Ungrounded answers: Review retrieval evidence, prompt/model changes, and evaluation samples. Unauthorized exposure: Restrict affected access, investigate audit evidence securely, and follow the incident response process. Interview takeaway: Production RAG monitoring combines end to end tracing, ingestion checks, SLO dashboards, quality sampling, actionable alerts, and incident response . A system can be fast and available while still returning stale or incorrect answers.

How do you continuously evaluate a production RAG system?

Continuous evaluation of production RAG is the repeatable cycle of collecting representative real world failures, updating evaluation datasets, testing changes, and monitoring whether deployed answers remain accurate, grounded, safe, and useful. Unlike one off offline or online evaluations, it connects production feedback to release decisions over time . Beginner friendly example An employee handbook assistant is evaluated before launch, but a new policy version is uploaded every week. Last week's successful answers do not prove today's answers are correct. A continuous evaluation loop: 1. Sample real questions safely, including those receiving weak citations, empty retrieval results, or user corrections. 2. Have reviewers add approved questions, expected evidence, and policy version labels to a maintained evaluation dataset. 3. Run retrieval only and answer level tests against the current index, prompt, and model versions. 4. Compare scores with the last accepted release. 5. Block or investigate harmful regressions, then deploy with safeguards and review new production outcomes. What should the process include? 1. Representative test cases: Cover frequently asked, difficult, multilingual, ambiguous, unanswerable, newly changed, and permission sensitive queries. Remove or minimize personal data. 2. Stable evaluation versioning: Record question, allowed evidence, expected behavior, source version, annotation guidelines, and dataset version. Keep a fixed regression set plus a changing recent traffic set. 3. Separate retrieval from generation: Test whether correct evidence is present and authorized before judging the final answer. Use measures such as evidence recall, citation support, groundedness, response relevance, and correct abstention. 4. Human calibration: Periodically compare automated/LLM judge decisions against trained human judgments; review disagreements and avoid treating judge scores as ground truth. 5. Evaluation triggers: Run suites on document updates, indexing or chunking changes, embedding/reranking model updates, prompt changes, new LLM versions, and a recurring schedule. 6. Release gates: Define a baseline and acceptable change thresholds by risk. Review critical failures—even if average scores improve. 7. Post release monitoring: Track sampled answer quality, feedback, retrieval failures, latency, and cost by query segment. Feed confirmed incidents back into the test set. 8. Governance and auditability: Store evaluation reports, configuration versions, reviewers, failure causes, and release/rollback decisions. Numerical example A team maintains 200 reviewed test questions . The old system correctly answers 178 , while a new release correctly answers 186 . Previous accuracy: 178 / 200 = 89% New accuracy: 186 / 200 = 93% Improvement: 4 percentage points However, if two of the new errors involve unauthorized disclosure, the release should not pass simply because aggregate accuracy increased. Evaluate security critical cases separately and investigate uncertainty in the sample. Common mistakes Re running a static benchmark forever while user questions and source documents change. Grading only final answers without diagnosing retrieval failures. Assuming unvalidated LLM as judge scores are objective. Optimizing to the same small test set until it stops representing production. Ignoring permissions, freshness, rare high impact failures, and rollback criteria. Interview takeaway: Continuous RAG evaluation is a versioned feedback loop : production examples → reviewed test cases → automated and human evaluation → release gate → production monitoring → refreshed tests. It is broader than any single offline test or live dashboard.

How do you balance retrieval quality, latency, and cost in production RAG?

Balancing retrieval quality, latency, and cost in production RAG means selecting a pipeline configuration that returns sufficiently relevant, authorized evidence within an acceptable response time and operating budget . The goal is not to maximize retrieval scores at any price—it is to meet explicit service and answer quality requirements. Beginner friendly example A support assistant can use three configurations: A — Fast: Retrieve 5 chunks, no reranker. Quality on a fixed evaluation set: 82% ; p95 end to end latency: 1.1 seconds ; estimated cost: $0.004/request . B — Balanced: Retrieve 20 candidates, rerank to 5. Quality: 91% ; p95 latency: 1.7 seconds ; cost: $0.008/request . C — Heavy: Retrieve 60 candidates, run an expensive reranker, pass 12 chunks to the model. Quality: 93% ; p95 latency: 3.2 seconds ; cost: $0.020/request . These are illustrative measurements, not universal benchmarks. If the product requires at least 90% evaluated quality, p95 below 2 seconds, and cost below $0.01/request , B satisfies all three. C gains only 2 percentage points while exceeding the latency and cost limits. Practical optimization workflow 1. Define what “quality” means: Use judged evidence recall or relevance, answer groundedness, citation correctness, task success, and permission correctness. Choose measurements appropriate to the use case. 2. Set hard constraints: Establish minimum acceptable quality, p95/p99 latency SLOs, security requirements, and a per request or monthly cost budget. 3. Create a baseline: On representative, versioned queries, measure both retrieval only quality and end to end answers. Report embedding, search, reranking, generation, cache, and infrastructure costs separately. 4. Vary one major parameter at a time: Test Top K, ANN search effort, BM25/vector fusion, metadata filters, reranker type and candidate count, chunk size, context length, and caching. Re evaluate after interacting changes. 5. Inspect the Pareto frontier: Prefer configurations where no alternative is at least as good on quality, at least as fast, and no more expensive , with a strict improvement in at least one dimension. 6. Use selective computation: Route simple questions through a light pipeline; use deeper retrieval or stronger reranking for difficult or high risk questions. Validate the routing logic so hard queries are not consistently under served. 7. Keep context concise: Retrieve enough candidates for recall, rerank intelligently, then send only relevant evidence to the generator. More retrieved candidates need not mean more prompt tokens. 8. Optimize safely: Reuse versioned embeddings, retrieval results, and eligible prompt prefixes when permissions and freshness allow; never trade tenant isolation or factual grounding for speed. 9. Monitor and revisit: Track quality regressions, p95/p99, tokens, cost/request, cache hit rate, source freshness, and traffic mix. Re benchmark after model, index, or document changes. Numerical cost example At 100,000 requests/month , the illustrative costs above yield: A: 100,000 × $0.004 = $400/month B: 100,000 × $0.008 = $800/month C: 100,000 × $0.020 = $2,000/month Those totals are simplified variable costs; include fixed database, GPU, storage, monitoring, and data transfer charges when calculating actual operating cost. Common mistakes Optimizing vector search speed while ignoring the slower LLM generation stage. Using only average latency and overlooking p95/p99 slow requests. Increasing Top K without checking whether the extra passages improve answer quality. Comparing configurations on different datasets or workload mixes. Treating lower cost as success despite lost grounding, citations, freshness, or authorization. Interview takeaway: Start with quality and safety gates , then choose the lowest cost configuration that meets latency SLOs . Benchmark the whole pipeline, inspect Pareto trade offs, use adaptive retrieval where warranted, and monitor performance as workloads change.

Your document ingestion job reports success, but 15% of the uploaded documents are missing from search. How would you investigate?

Production scenario: A RAG ingestion dashboard shows SUCCESS , but users cannot find 15% of the documents they uploaded. Treat this as a data completeness incident , not automatically as a vector search relevance problem. A successful orchestration run may mean only that the overall job completed—not that every document became searchable. 1. Define what “missing” means First distinguish two failures: Not indexed: The expected document/version or its chunks never reached the active searchable index. Indexed but not retrieved: The document exists, but exact lookup, tenant filters, index routing, or relevance ranking keeps it out of results. Use a document ID/version lookup , not only a semantic question, to test presence. Check the correct tenant, collection, index alias, and access scope. 2. Reconcile counts using an ingestion manifest Give each source document a stable tenant id + document id + source version, and record an explicit state for each stage: 1. Discovered from source / upload manifest 2. Fetched and checksum verified 3. Parsed and validated 4. Chunked with expected chunk count 5. Embedded with success/error information 6. Written to storage or vector database 7. Searchable in the active index after visibility/refresh 8. Failed or quarantined , with reason and retry history Compare distinct document IDs at each stage; chunk totals alone cannot prove document coverage . Preserve zero text documents as explicit parse failures or known exceptions instead of silently treating them as success. 3. Root cause debugging checklist Discovery or ingestion selection: Were uploads excluded by file type, size, folder rules, date window filters, or pagination errors? Inspect the upload manifest and source connector cursors. Parsing: Look for password protected PDFs, scans without OCR, corrupt files, timeouts, language/encoding errors, or documents yielding zero usable text. Chunking and embeddings: Identify empty chunks, unsupported token lengths, per item batch failures, embedding API throttling, and partial responses. Queue or workers: Examine pending/in flight messages, retry counts, visibility/acknowledgment ordering, worker crashes, and dead letter queues. A batch level success log may hide individual failures. Queue delivery counts are not unique document completion counts. Vector writes: Inspect per document/per vector upsert acknowledgments, payload validation, namespace mismatches, inconsistent embedding dimensions, write failures, and index refresh or eventual consistency lag. Search visibility: Check active index alias, tenant/ACL filters, stale replicas, and document version routing. Compare exact ID retrieval with filtered keyword or vector search. Silent exception handling: Look for except: pass, ignored batch errors, or a job marked successful despite nonzero per document failures. Cloud/search tooling can show “succeeded” runs even when the expected documents were not added, so document level reconciliation is essential (see AWS Kendra troubleshooting guidance: https://docs.aws.amazon.com/kendra/latest/dg/troubleshooting data sources.html). 4. Beginner friendly numerical investigation Assume 1,000 uploaded documents , but exact ID checks find only 850 searchable . Coverage: 850 / 1,000 × 100 = 85% Missing: 1,000 850 = 150 documents A hypothetical reconciliation report shows: 1,000 discovered 980 fetched ( 20 missing at fetch ) 940 parsed ( 40 failed at parsing ) 910 embedded ( 30 failed at embedding ) 870 written ( 40 failed at index write ) 850 searchable ( 20 awaiting refresh, misrouted, or filtered ) These stage differences sum to 150 , provided counts refer to the same document/version cohort and each document is counted once. The figures are illustrative, not observed MachinoAI production data. Important: A newly indexed record might be invisible briefly due to index refresh; distinguish normal bounded lag from a persistent missing document incident. 5. Engineering fix and safe recovery 1. Capture a per document failure report with stage, retry count, error code, source version, tenant, and index target. 2. Fix the failed stage rather than indiscriminately re embedding all 1,000 documents. 3. Retry only missing/failed document versions using idempotent upserts and stable document/chunk IDs; avoid duplicate records on redelivery. 4. Use bounded retry with backoff for transient failures; send repeatedly failing items to a dead letter queue for inspection and controlled replay. 5. Mark a document SEARCHABLE only after a document/version presence check in the active search path. A write acknowledgment alone is not proof of searchability. 6. Reconcile again and perform targeted authorized retrieval smoke tests. Close the incident only when unresolved exceptions are accounted for. If the same upload can be processed concurrently, guard its state transition with a version check or compare and swap so an older run cannot overwrite a newer version. 6. Prevention and production monitoring Alert on document coverage , not just job success: searchable distinct expected document versions / eligible source document versions. Track ingestion stage failures, queue age, dead letter count, index refresh lag, and document level completion latency by source/tenant/file type. Define clear policies for intentionally excluded or quarantined files and report them separately. Use a periodic source to index reconciliation job and canary documents with known exact ID and retrieval assertions. Avoid logging sensitive document text; use safe IDs and redacted error details. Trade off: Verifying searchability for every document increases write/read overhead, but detecting missing content before users complain is often worth it. Large systems can use batched or sampled read checks plus full manifest reconciliation to control cost. Interview takeaway: Job SUCCESS is not document completeness. Trace each document across discovery → parsing → embedding → indexing → search visibility, identify the exact failed stage, replay safely, and gate operational success on reconciled coverage rather than a green pipeline dashboard. Further reading: AWS Kendra troubleshooting (https://docs.aws.amazon.com/kendra/latest/dg/troubleshooting data sources.html) and Amazon SQS operational metrics (https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs available cloudwatch metrics.html).

A connector stops syncing documents without raising any exceptions. How would you detect this silent failure?

Production scenario: A SharePoint/Drive connector reports RUNNING or even SUCCESS, yet newly edited documents stop appearing in a RAG knowledge base. No exceptions fire because the worker is alive and its requests succeed, but the data is no longer progressing . The key distinction is liveness (the process is alive) versus freshness and progress (source changes have reached searchable storage). 1. What would you detect first? 1. Last successful sync versus last useful sync: Record the time the scheduler last completed, last new/updated source event observed , last committed destination checkpoint, and last document made searchable. 2. Expected source activity: Compare new source changes with changes detected by the connector. A zero record sync can be perfectly healthy for an unchanged source; don't alert on zero output without checking upstream activity or source change expectations. 3. Cursor progression: Track the source watermark, page/continuation token, CDC offset, and committed checkpoint. A cursor that never advances while the source changes is suspicious. 4. End to end freshness: Measure searchable at source updated at for each eligible document. Alert when the p95 or oldest pending source change exceeds your source specific freshness SLO. 5. Heartbeat plus progress: A heartbeat detects dead workers; a progress watchdog checks whether meaningful work has moved forward. They catch different failure modes. For example, Airbyte documents heartbeats for unresponsive sources and destinations, while Debezium describes both source lag and time since last event signals. A heartbeat alone cannot prove that expected source changes reached your vector index. 2. Root cause investigation Start by reproducing with one known, authorized canary document , updating it at the source and tracing its version. Scheduler and worker: Was the job scheduled, claimed, started, and completed? Is the worker merely heartbeating while stuck on pagination or backoff? Credentials and permissions: Has a token scope narrowed, a folder moved, or a service account lost access? Some APIs return valid empty pages for inaccessible or newly excluded content. Incremental sync state: Was a cursor accidentally advanced past records? Are timestamps rounded, timezones misinterpreted, or updates sharing the same cursor value omitted because the query uses instead of a safe tie breaking strategy? Pagination and API quotas: Are continuation tokens ignored, expired, or cyclic? Are 429 responses handled with bounded retries rather than silently treated as empty results? Filtering and transforms: Did a folder filter, MIME allowlist, deduplication rule, or parsing change start dropping all incoming records? Queue, destination, and indexing: Are source records observed but not acknowledged downstream? Check queue backlog, destination commits, dead letter counts, document versions, and active index visibility. Status reporting: Is SUCCESS set at the job level despite zero processed records, missing partitions, or uncommitted state? Important: Never blindly reset or skip a source cursor before recording the last safely committed checkpoint and affected time range. You could create gaps or duplicates. 3. Beginner friendly numerical example A company expects SharePoint changes to become searchable within 30 minutes . At 10:00, the source contains 120 updated documents since the 09:00 checkpoint, but the connector reports successful syncs at 09:15, 09:30, 09:45, and 10:00 with zero records loaded . Source changes observed independently: 120 . Connector changes loaded: 0 . Unreplicated changes: 120 0 = 120. Oldest outstanding update at 09:05 is 55 minutes old at 10:00, exceeding the 30 minute freshness target by 25 minutes . These figures are illustrative. The independent source count and connector count must refer to the same scope, change window, and treatment of updates/deletes; raw event totals may not equal distinct document versions. 4. Engineering solution 1. Persist a durable sync run ledger keyed by connector, source partition, cursor range, and run ID. Record examined records, emitted records, destination acknowledgments, committed checkpoints, and observed maximum source modification time. 2. Run a separate watchdog that compares source side modification counts or a known canary with destination/index progress. Do not depend solely on worker reported SUCCESS. 3. Expose metrics such as seconds since last successful sync, seconds since last committed progress, source to index lag seconds, cursor stalled, records emitted, records committed, consecutive empty syncs, and dead letter count. 4. Create source aware alert rules : an empty run is fine on a quiet source; a stale cursor with confirmed upstream changes, prolonged queue age, or a missing canary is not. 5. Recover using bounded overlap windows or a documented resnapshot/backfill starting at the last destination acknowledged checkpoint. Use stable document IDs, source versions, and idempotent writes to tolerate redelivery. 6. Validate repairs by comparing a source manifest with indexed document/version IDs and testing authorized retrieval—not by checking connector status alone. 5. Trade offs and prevention Polling source changes provides a strong independent signal but consumes API quota. Use sampled partitions, adaptive cadence, or event/CDC signals when available. Canary files detect breaks in the whole path but won't prove coverage for all folders, tenants, or file types. Combine them with periodic reconciliation. Aggressive no progress alerts generate false positives on quiet sources and during approved maintenance. Use per source activity baselines, thresholds, and suppression windows. Checkpoint safety: Advance the durable cursor only after the destination has acknowledged the relevant records. This prevents silent data loss across crashes. Maintain connector dashboards, on call ownership, runbooks, and scheduled sync resumption drills. Interview takeaway: A connector can be alive, exception free, and still stale . Monitor independent source activity, committed cursor progress, and source to search freshness; alert on unexplained lack of progress, then recover safely from a verified checkpoint. References (additional research): Airbyte — Heartbeats: https://docs.airbyte.com/platform/understanding airbyte/heartbeats Airbyte — Protocol and state/checkpointing: https://docs.airbyte.com/platform/understanding airbyte/airbyte protocol Debezium — Monitoring source lag and time since last event: https://debezium.io/documentation/reference/stable/operations/debezium platform.html

Your ingestion workers process the same message multiple times. How would you prevent duplicate vectors and inconsistent document versions?

Production incident: A RAG ingestion queue delivers the same “index document” event to two workers. Both generate embeddings and write vectors. The search index now contains duplicate chunks, or an older document version overwrites a newer version. This is usually a message processing idempotency and concurrency problem , not simply a duplicate file problem. Why can the same event execute more than once? At least once delivery: Many queues intentionally redeliver messages if acknowledgments are lost or delayed. Visibility timeout / lease expiry: Worker A is still embedding when its lease expires; Worker B receives the same message. Crash between write and acknowledgment: The vector upsert succeeds, but the worker dies before acknowledging the message. Producer retries: A producer submits the same logical update more than once, sometimes with different queue message IDs. Out of order versions: An event for version 7 finishes after an event for version 8. AWS documents at least once SQS delivery and redelivery when visibility timeouts expire; a queue level deduplication feature does not replace safe application writes. Step by step root cause investigation 1. Pick one affected tenant id/document id and inspect the source version, queue message IDs, delivery attempts, worker IDs, processing start/end times, and acknowledgment records. 2. Query the vector index by document ID and version . Compare its chunk IDs/count with the expected chunk manifest. 3. Check whether write operations used random vector IDs (duplicate inserts) or deterministic IDs (safe updates for identical content). 4. Search worker traces for lease expiration, retry storms, lost acknowledgments, or two workers handling the same document concurrently. 5. Compare the persisted active version with message versions to detect stale workers overwriting newer data . 6. Reproduce under controlled fault injection: pause a worker after its vector write but before acknowledgment, then redeliver the event. Engineering solution: design for repeated delivery 1. Stable operation identity: Derive an idempotency key from tenant id + document id + source version + pipeline version. Use an authoritative version or content hash; do not rely only on the queue message ID . 2. Claim work atomically: In a durable job ledger, create or claim a unique processing record. Mark duplicates as already processing/completed when appropriate. Expired claims must be recoverable. 3. Deterministic vector IDs: Use a stable chunk identity, for example hash(tenant id, document id, source version, chunk index, chunker version). Write with upsert rather than insert with new random ID. 4. Fence stale versions: Track latest accepted source version and an active index generation. Before activating a version, atomically verify that it is still current. A worker finishing version 7 must not make it active after version 8 has won. A lock alone is insufficient if an expired worker can continue writing; use a monotonic fencing token or transactional compare and swap. 5. Publish atomically at the logical level: Stage a document version's chunks under a generation/version; switch the active pointer only after expected chunks are present and searchable. Queries must filter to the active version. This avoids partially updated results even if vector writes are not transactional. 6. Acknowledge only after durable completion: Commit the document state after the required downstream writes/verification. Then acknowledge the queue message. A retry following an acknowledgment failure should detect the completed operation and acknowledge without reindexing. 7. Safe cleanup: Reconcile abandoned generations and delete superseded chunk IDs only after the new active version is safely available. 8. Recovery: Apply bounded retry/backoff and dead letter handling. Investigate poison messages instead of retrying indefinitely. Numerical example Suppose 1,000 unique document update events are delivered 1,150 times due to retries and lease expiry. Each update produces 20 chunks . With random IDs on every attempt: 1,150 × 20 = 23,000 vector rows may be inserted. With one committed generation per logical event: 1,000 × 20 = 20,000 intended vector rows. Potential excess: 3,000 rows, or 15% more rows than intended . These values are illustrative and assume each event represents a distinct accepted version and has 20 chunks. In a real system, several versions of one document may exist temporarily during a safe cutover. Minimal processing sketch Treat this as pseudocode, not copy paste production code: actual implementations must handle lease renewal, transactions, and failures between operations. Trade offs and prevention Stable IDs + upserts reduce duplicate writes, but do not alone prevent old versions from replacing new ones. Per document serialization simplifies ordering but limits throughput for frequently updated documents; version fencing supports higher concurrency with additional complexity. Staging and active version pointers add storage and bookkeeping but greatly reduce partial index visibility. Queue exactly once settings may reduce deliveries; they do not automatically make external vector/database side effects exactly once. Monitor duplicate delivery rate, idempotency key conflicts, stale version rejections, orphaned generations, expected vs actual chunk counts, redelivery attempts, lease expirations, and dead letter counts. Add a load test that deliberately kills a worker after a successful vector write and before acknowledgment . Interview takeaway: Assume messages can repeat. Combine business level idempotency keys, deterministic vector upserts, durable work state, version fencing, and active generation cutover so repeated or out of order processing never corrupts the searchable corpus. References: AWS SQS — At least once delivery: https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/standard queues at least once delivery.html AWS SQS — Visibility timeout: https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs visibility timeout.html Google Cloud Pub/Sub — Exactly once delivery and limitations: https://docs.cloud.google.com/pubsub/docs/exactly once delivery

A 2 GB PDF causes ingestion workers to run out of memory. How would you redesign the processing pipeline?

Production incident: A customer uploads a 2 GB PDF to a RAG platform. A worker with a 4 GiB memory limit loads the entire file, renders pages for OCR, accumulates extracted images and text, and is terminated by the container's out of memory (OOM) killer. The job is retried and crashes again. The solution is bounded memory document processing , not simply assigning a larger machine. 1. Diagnose the memory failure before redesigning Confirm OOM: Inspect Kubernetes OOMKilled/exit code 137, container memory peaks, process RSS, restart count, and the worker's memory limit. Distinguish OOM from CPU timeouts or disk exhaustion. Locate the allocation spike: Add spans and memory measurements around download, PDF open, page rendering, OCR, image extraction, text aggregation, chunking, and embedding batches. Inspect document structure: Record page count, scanned versus selectable text pages, oversized images, embedded attachments, encryption, and malformed objects. A 2 GB file does not imply a 2 GB in memory footprint; decoded images and render buffers can be much larger. Check concurrency: Two simultaneous PDFs each using 2 GiB can exhaust a 4 GiB worker even when each file succeeds alone. Reproduce safely: Use a staging worker with realistic resource limits and a sanitized or approved copy of the problematic file; record the exact stage and peak RSS. 2. Redesign ingestion as a bounded memory pipeline 1. Store the upload directly in object storage. Stream the upload to a durable blob and enqueue only document id, object URI, checksum, version, and tenant scope. Avoid passing a 2 GB payload through the job queue. 2. Preflight: Validate MIME/type, size, basic PDF health, page count, encryption, page dimensions, and safe processing limits. Quarantine suspicious inputs instead of retrying them endlessly. 3. Process by page or small page ranges. Open and read only the necessary ranges; extract text and metadata; write page artifacts incrementally to storage. Close PDF/page/render objects between batches and release library caches when supported. 4. Render and OCR selectively. OCR image only or text poor pages rather than blindly rendering every page. Bound DPI, pixel count, image dimensions, per page time, and total OCR budget. 5. Chunk incrementally. Keep only a bounded text window plus enough overlap to avoid splitting meaningful sections. Retain page IDs, coordinates, and document lineage. 6. Batch embeddings with size limits. Submit small bounded chunks/tokens per request; persist generated vectors and checkpoint completed page ranges. Never hold all vectors, images, and chunks in a single array. 7. Isolate risky parsing. Run PDF extraction/OCR in resource limited subprocesses or dedicated workers separate from critical queue/index services. Recycle workers that leak native memory. 8. Publish only after verification. Stage chunks under a document version generation; check expected page/chunk counts and error flags before activating the version in the searchable index. Important nuance: Some PDF parsers need random access to a seekable file, so “stream everything through one forward only iterator” is not always feasible. A local temporary file or bounded page range subprocess can be more robust than assuming true streaming is supported. 3. Beginner friendly numerical example Suppose a 2 GB PDF has 10,000 pages . Rendering a page at 2,000 × 3,000 RGB pixels uses approximately: 2,000 × 3,000 × 3 = 18,000,000 bytes ≈ 18 MB Holding 100 rendered pages at once uses roughly 1.8 GB just for raw pixel buffers. Holding 5 pages uses roughly 90 MB in raw pixels. OCR engines, parser caches, decoded embedded images, embeddings, and process overhead require additional RAM; 90 MB is not total worker memory . A 5 page batch creates 2,000 checkpointable batches for 10,000 pages. If a worker dies after batch 1,250, a checkpointed design can resume from the next uncommitted batch rather than restarting every page. Chunk overlap and idempotent page version IDs are necessary to avoid gaps or duplicates. 4. Recovery after an OOM 1. Stop automatic retry storms and isolate the offending job in a large document queue with lower concurrency. 2. Preserve the source blob and ingestion manifest. Verify the last durably completed page range; do not trust an in memory counter. 3. Resume with smaller batches, reduced OCR resolution where quality permits, and appropriate parser resource limits. 4. Use deterministic chunk IDs and idempotent upserts so replay cannot create duplicate vectors. 5. Quarantine the file with an actionable error if limits are exceeded repeatedly; provide a human review or approved offline processing path. 6. Validate searchable document coverage and sample citations to pages near batch boundaries before marking the ingestion complete. 5. Trade offs and prevention Smaller batches: Lower peak RAM and better recovery granularity, but more I/O, checkpoint writes, and worker overhead. Selective OCR: Saves CPU/memory, but quality checks must catch text embedded in diagrams or mixed scanned pages. High DPI: Helps small text in some documents, but memory grows with rendered pixel area; adapt DPI based on page content. Process isolation: Protects the platform from parser crashes but adds startup and interprocess transfer costs. Temporary files: Bound RAM and permit random access but require secure encrypted scratch storage, cleanup, and disk quotas. Observability: Monitor per file peak RSS, OOM count, pages/sec, OCR hit rate, temporary disk use, retries, queue age, failed page ranges, and source to index completeness. Production guardrails: Set limits on pages, rendered pixels, time per page, total compute, worker concurrency, extracted attachment size, and uncompressed expansion. Test with large scans, compressed images, malformed PDFs, and mixed text/image documents. Interview takeaway: Do not load or render a giant PDF all at once. Use object storage ingestion, bounded page range extraction, selective OCR, incremental chunk/embedding batches, isolated worker processes, durable checkpoints, and versioned publication . Measure actual decoded memory usage rather than inferring RAM from file size. References: PyMuPDF large file FAQ: https://pymupdf.readthedocs.io/en/latest/faq/index.html ; Apache Tika PDF parser resource/OCR configuration: https://tika.apache.org/docs/4.0.x/configuration/parsers/pdf parser.html ; Apache Tika parser robustness: https://github.com/apache/tika/blob/main/docs/modules/ROOT/pages/advanced/robustness.adoc

An ingestion job crashes after generating embeddings but before updating the document-processing status. How would you recover safely?

Production incident: A RAG worker successfully generates embeddings for a 300 chunk policy document, then crashes before changing the job state from EMBEDDING to EMBEDDED . The queue retries it. The status database says work is incomplete, but embeddings may already exist in an external cache or vector store. The key question is not “should we retry?” but “which side effects were durably completed before the crash?” A status field is not sufficient evidence that an external operation succeeded or failed. 1. Diagnose the exact crash boundary 1. Locate the job using tenant id, document id, source version, pipeline version, and attempt id. 2. Correlate traces: parsing completion, chunk manifest, embedding API response, embedding artifact writes, vector upserts, DB status transaction, and queue acknowledgment. 3. Distinguish three cases: Vectors computed but not persisted: Only the worker saw them; recomputation is necessary unless a provider side reusable cache is available. Embedding artifacts persisted, status update missed: Reuse validated vectors and complete the missing state transition. Vectors already upserted to the search index: Verify exact chunk IDs and version, then reconcile status rather than blindly inserting them again. 4. Look for a killed worker, expired lease, DB outage, transaction rollback, or crash immediately before status commit. Verify whether a second worker has acquired the same job. 5. Check whether a newer source version is already active. Recovery must not make an obsolete version searchable. 2. Engineering solution: checkpoints + reconciliation Stable operation key: hash(tenant id, document id, source version, chunker version, embedding model version). Durable manifest: Store expected chunk IDs, content hashes, embedding dimensions, model identifier, and expected count before processing. Checkpoint embeddings: Persist each completed batch to durable artifact/object storage or a suitable versioned vector staging area. Record content hashes and batch completion. A local worker memory result is not a checkpoint. Use idempotent writes: Generate deterministic IDs such as hash(job key, chunk index); replaying a completed batch must update/reuse the same logical vectors rather than create random duplicates. Reconcile before retry: Compare the manifest with the durable embedding cache/staging index. Recompute only missing or invalid batches. Fence concurrent workers: Use an atomic job claim with an expiring lease and monotonically increasing fencing token. Commit job state only if the lease/version is still current. Activate after verification: Mark the document version searchable only once the expected vectors are present, valid, authorized, and visible on the active retrieval path. Use a version/generation pointer to prevent exposing partial results. Acknowledge last: After durable completion and state commit, acknowledge the queue delivery. If the ack is lost, a later delivery should observe COMPLETED and simply acknowledge. Crash windows matter: There is generally no single transaction spanning a remote embedding provider, vector database, status DB, and queue. An outbox or workflow state machine can make local DB state and event publication atomic, but external side effects still require reconciliation and idempotency. AWS's transactional outbox guidance explicitly notes consumer deduplication, and SQS documents redelivery after timeouts. 3. Beginner friendly numerical example A document yields 300 chunks , embedded in batches of 50 : six batches total. Each batch is assumed to cost $0.005 in embedding charges (illustrative). Batches 1–4 completed and were durably checkpointed: 200 vectors . Batch 5 was returned by the provider but the worker crashed before saving it : 50 vectors of uncertain durable status . Batch 6 was never started: 50 vectors . On recovery, validate the checkpoint for 1–4, then recompute batches 5–6: 100 vectors , or two batches . A full restart would repeat all six batches, costing an additional $0.030 ; the checkpointed recovery costs $0.010 in new embedding calls. Illustrative avoided recomputation: four batches × $0.005 = $0.020 , not counting storage and validation overhead. The original provider call for batch 5 may already have been billed, even if its result was lost. Caching/checkpointing reduces future work; it does not retroactively undo provider charges. 4. Simple recovery pseudocode This is architectural pseudocode: actual code needs atomic lease acquisition, bounded retries, and checks appropriate to the vector database's consistency model. 5. Trade offs and prevention Frequent checkpoints: Reduce wasted embedding calls but increase storage operations and metadata overhead. Artifact cache vs direct vector upsert: Caching embeddings separately simplifies retry/reindex, but consumes storage; direct idempotent upserts may suffice for simpler systems. Single document locking: Limits duplicate work but reduces concurrency for rapidly updated documents; fencing avoids stale completion. Eventual index visibility: A successful vector write might not be searchable immediately. Use bounded polling/backoff; do not falsely fail a valid job because refresh is delayed. Orphan cleanup: Periodically remove staged generations no longer referenced by an active document version. Prevention checklist: Track jobs stuck between EMBEDDING and EMBEDDED, source to index lag, missing checkpoint batches, lease expirations, retry count, duplicate vector IDs, orphaned generations, and stale version activation attempts. Add fault injection tests at every boundary : after embedding response, after checkpoint write, after vector upsert, after status commit, and before queue acknowledgment. Interview takeaway: A crash after embedding but before a status update creates an uncertain commit problem . Recover by inspecting durable artifacts, reconciling against a chunk manifest, recomputing only missing work, fencing stale workers, and confirming search visibility before marking the document complete . References: AWS SQS at least once delivery: https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/standard queues at least once delivery.html AWS SQS visibility timeout: https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs visibility timeout.html AWS transactional outbox pattern: https://docs.aws.amazon.com/prescriptive guidance/latest/cloud design patterns/transactional outbox.html