Retrieval-Augmented Generation

Retrieval-augmented generation connects a generative model to a searchable knowledge source. Instead of relying only on model weights, the system fetches relevant documents and places them into the prompt or context.

Your SharePoint connector successfully syncs documents, but some permissions are outdated. How would you detect and resolve the inconsistency?

Production incident: A SharePoint to RAG connector reports SUCCESS because document content is synchronized. Yesterday, an employee lost access to a confidential folder, but the RAG index still lists their group in allowed groups. The user can retrieve sensitive passages that SharePoint itself now denies. This is an authorization staleness incident , not evidence that the document parser or embedding model failed. 1. Contain the risk first 1. Temporarily exclude affected document IDs, sites, or permission scopes from RAG retrieval until authorization can be trusted. Fail closed , not open, when permissions are unknown. 2. Invalidate tenant/user scoped retrieval caches, stored snippets, and answer caches involving the affected content. 3. Preserve privacy safe audit traces for incident response. Check whether revoked users actually received passages, citations, summaries, or answers. 4. Follow the organization's security incident process if unauthorized exposure occurred. Do not wait for a routine nightly content sync to restore access safety. 2. Investigate the permission mismatch Find the source of truth: For each affected file, determine its effective SharePoint access, including folder/site inheritance, unique file permissions, sharing links, and Microsoft Entra ID group membership. Compare source and destination: Capture a source ACL snapshot and the index ACL version or hash. Diff allowed user ids, allowed group ids, inheritance state, tenant/site/document IDs, and refresh time. Inspect the connector's change feed: A content modified at timestamp may not change when a permission changes. Verify whether the connector tracks ACL specific changes, inheritance changes, group changes, and deletions. Inspect permission propagation: Trace from Microsoft Graph through the connector queue, ACL resolution, metadata update, search index refresh, and cache invalidation. A successful content upsert is not proof that authorization metadata was refreshed. Check effective identities: Compare stable Entra object IDs, group membership at query time, nested/dynamic groups, and removed accounts. An email/name is not a reliable authorization key. Test both directions: Revocation must stop access; legitimate new grants should become available without violating the policy's freshness SLO. SharePoint specific note: Microsoft Graph's driveItem: delta supports sharing change discovery. For suitable SharePoint/OneDrive for Business scanning workflows, Prefer: hierarchicalsharing or Prefer: deltashowsharingchanges can help identify changes, subject to documented prerequisites and permissions. Delta alone is not a complete replacement for verifying effective ACLs and group memberships. 3. Beginner friendly numerical example A RAG corpus holds 10,000 SharePoint documents . A reconciliation job checks ACL versions/hashes and finds 300 mismatches , of which 120 contain removed user or group grants. ACL mismatch rate: 300 / 10,000 × 100 = 3%. Potential overexposure candidates: 120 / 10,000 × 100 = 1.2%. If the access control refresh target is 15 minutes , but some revoked grants remain searchable after 90 minutes , their age exceeds the target by 75 minutes . These counts are illustrative. A stale grant is a serious potential exposure, but proving actual leakage requires reviewing the relevant retrieval and answer audit records. 4. Engineering solution 1. Track content and ACL versions separately. Model source document version and source acl version independently, with a last validated timestamp and stable source IDs. 2. Build an ACL only update path. When permissions change, update searchable ACL metadata or a secure policy reference without re parsing, rechunking, or re embedding unchanged content . 3. Normalize principal identities. Use stable Entra user/group object IDs and represent inheritance explicitly; resolve effective grants according to the integration's supported SharePoint permission model. 4. Enforce restrictions before evidence reaches the LLM. At query time, authorize document candidates using trusted user claims and verified group memberships. Do not rely on prompting the model to ignore restricted passages. 5. Handle revocations promptly. Process access removals with higher priority than ordinary content updates. If freshness cannot be guaranteed for sensitive content, revalidate against an authoritative permission service or quarantine the affected documents until reconciliation finishes. 6. Use safe, ordered updates. Apply document ID + ACL version checks so an old synchronization task cannot overwrite a newer restrictive ACL. If a permission update fails, do not silently retain access. 7. Invalidate caches. Cache keys must include an appropriate authorization scope/version. Clear already cached evidence after a revocation event. 8. Reconcile regularly. Compare sampled or full source ACL manifests against index metadata and run synthetic positive/negative access tests for representative users and groups. 5. Trade offs and operational prevention Frequent polling improves ACL freshness but consumes Microsoft Graph quota and operational capacity. Events/delta based sync reduce unnecessary work but still require checkpoint recovery, gap detection, and periodic reconciliation. Materialized ACLs in the search index give efficient server side filtering but can be stale; live source authorization provides fresher checks at additional latency and availability cost. Group expansion at ingestion can become stale after membership changes; query time membership validation or a dedicated authorization service reduces this risk. Fail closed retrieval may temporarily hide legitimate content, but is preferable to exposing restricted data during a security sensitive incident. Monitor: last successful ACL sync, source to index ACL lag, mismatch count, failed revocation updates, group resolution errors, unauthorized result tests, cache invalidations, and permission denied events. Run a canary that revokes access to a test document and verifies the retrieval system stops serving it within the agreed policy window. Interview takeaway: Content synchronization and permission synchronization are different guarantees. Detect drift by comparing effective SharePoint ACLs with indexed ACL metadata, prioritize revocations, enforce trusted query time authorization, invalidate caches, and verify access denial behavior—not merely connector job success. References: Microsoft Graph — DriveItem delta and permission change scanning: https://learn.microsoft.com/en us/graph/api/driveitem delta?view=graph rest 1.0 Microsoft ISE — Propagating SharePoint Document Permissions to AI Search and RAG Pipelines: https://devblogs.microsoft.com/ise/sharepoint doc level access/

A document update arrives while its previous version is still being indexed. How would you prevent race conditions?

Production incident: A policy PDF version 12 starts indexing at 10:00. At 10:02, the source publishes version 13. The version 13 worker finishes first and makes the new policy searchable, but the slower version 12 worker completes afterward and replaces it. Users now receive older answers even though the latest update was successfully processed . This is an out of order completion race . Unlike duplicate delivery of the same message, two legitimate updates to the same document are executing at the same time. 1. Diagnose the race 1. Find the affected tenant id + document id and list the source's authoritative revisions, ingestion events, worker attempts, index generations, and activation timestamps. 2. Create a timeline of event received → parsing → embedding → index write → activation for versions 12 and 13. 3. Compare source version ordering with completion ordering . Completion time is not proof of freshness. 4. Check whether updates write the same chunk IDs, delete each other's chunks, or use an unconditional UPDATE active version = .... 5. Verify what the retriever actually filters on: active generation, source version, or merely whatever vectors remain in a shared collection. 6. Look for additional races between permission revocations, document deletion, and content updates; a deleted or inaccessible document must not be resurrected by an old worker. 2. Design a safe versioned pipeline Use stable identity: Keep tenant id + source document id stable; assign each source update an authoritative, monotonically comparable revision where supported. If sources expose only timestamps or change tokens, define ordering explicitly and handle ties, clock skew, and deletes. Stage each revision independently: Store chunks under immutable document id + source version + pipeline generation. Do not let in flight workers overwrite the current generation. Track desired and active revisions: A small authoritative state record contains desired version, active version, active generation, and an optional fencing token. Coalesce obsolete work: When version 13 arrives, mark version 12 superseded; cancel expensive queued work if practical. Cancellation is an optimization, not the correctness guarantee. Verify before activation: Confirm expected chunk count, correct embedding model, source metadata, ACLs, and search visibility for version 13. Activate with an atomic conditional update: Promote the staged revision only if it is still the latest desired version and the activation token is valid. A worker for version 12 must fail after the desired version advances to 13. Retrieve only the active generation: Pass tenant and access restrictions plus active generation to retrieval. In systems without index level atomic aliases, use an authoritative manifest or routing layer to prevent mixed version results. Clean up asynchronously: After successful cutover and an appropriate safety window, remove obsolete staged vectors. Keep rollback artifacts only as long as policy permits. 3. Beginner friendly numerical example Two workers process a policy containing 100 chunks : 10:00: Version 12 begins indexing; estimated duration 8 minutes . 10:02: Version 13 is received and becomes desired version=13; processing takes 3 minutes . 10:05: Version 13 finishes, passes validation, and is activated. 10:08: Version 12 finishes but its conditional activation fails because 12 != desired version(13). Correct result: Users see 100 version 13 chunks , not 100 old chunks or an accidental mixture of 200 chunks. Both versions might temporarily exist in staging, but only the authorized active generation is queried. If version 13 fails validation , version 12 should not silently overwrite it. Follow the business freshness policy: either retain an explicitly identified previously approved version, or temporarily withhold the document when outdated information is unsafe. 4. Simplified control logic The compare and swap operation must be implemented transactionally against the authoritative state store. If worker leases can expire, use fencing tokens to prevent old holders from committing. A vector database's unconditional upsert alone cannot provide this guarantee across independent data stores. 5. Trade offs and prevention 1. Serialize per document: Simple ordering, but a slow update delays newer versions and creates backlog. 2. Concurrent processing + conditional activation: Better freshness under load, but requires extra generation storage and reliable state transitions. 3. Cancel stale jobs: Saves embedding costs, but cancellation can race with a worker already writing; version checks remain mandatory. 4. Alias/pointer cutover: Helps avoid partial reads, but index visibility and cache invalidation must be tested for the specific search engine. 5. Rollback: An older version may be intentionally restored only through a new, audited revision or explicit administrative transition—not by a stale worker finishing late. Monitor superseded job count, stale activation attempts, desired versus active version lag, mixed version retrieval, orphaned generations, and indexing failures . Add fault injection tests where the older job finishes last, a worker restarts mid write, and a delete/revoke event arrives during indexing. Interview takeaway: Do not let worker completion order determine document truth . Stage revisions separately, record the latest desired version, activate only with an atomic version check , filter retrieval to the active generation, and retire obsolete work safely. Further reading: PostgreSQL transaction isolation and conditional updates (https://www.postgresql.org/docs/current/transaction iso.html); AWS DynamoDB conditional updates (https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/Expressions.ConditionExpressions.html).

Your ingestion queue grows from 1,000 to 500,000 documents overnight. How would you manage backpressure, priorities, and worker scaling?

Production incident: Overnight, the RAG ingestion backlog rises from 1,000 to 500,000 documents . Search still works, but newly uploaded documents remain unsearchable for hours. Simply adding workers can overwhelm OCR services, embedding API quotas, vector database writes, and memory. Treat this as a capacity, admission control, and freshness incident . 1. Diagnose the backlog before scaling 1. Measure arrivals and completions: Track incoming documents/minute, successfully indexed documents/minute, retries, failures, oldest message age, and estimated time to drain. Distinguish queue messages from unique document versions. 2. Find the bottleneck stage: Trace fetch → parse/OCR → chunk → embed → vector upsert → search visibility. Record per stage throughput, p95 latency, queue depth, worker utilization, and external 429/timeouts. 3. Check workload mix: A 2 KB text note and a 500 page scanned PDF are not equivalent jobs. Measure pages, OCR pixels, embedding tokens, and estimated work units. 4. Inspect downstream limits: CPU/RAM, GPU memory, embedding tokens/minute, database write QPS, network bandwidth, and tenant quotas. Extra workers may worsen throttling. 5. Separate urgent from bulk work: Security revocations and important document updates may require faster treatment than a historical migration of hundreds of thousands of files. 2. Engineering solution: backpressure and fair scheduling Bound admission: Limit in flight jobs, OCR processes, embedding batches, and vector upsert concurrency. If a downstream dependency saturates, reduce intake/concurrency rather than creating unlimited retries. Apply explicit backpressure: Pause or throttle low priority producers, limit connector polling, or stage uploads in object storage for later processing. Accept new uploads with a clear QUEUED status rather than falsely marking them searchable. Use priority lanes: Separate urgent document deletion/revocation, current incremental updates, and bulk historical backfills. Assign reserved capacity or weighted fairness to avoid starving bulk work indefinitely. Enforce tenant fairness: Cap per tenant concurrency and token consumption. A single customer's mass import must not monopolize the entire ingestion fleet. Split expensive stages: Independently scale OCR, extraction, embeddings, and index writing workers. Use dedicated queues and quotas where the architecture supports them. Autoscale from workload, not just raw queue size: Consider backlog per worker and queue waiting age, adjusted for average processing time. AWS recommends backlog per task metrics for SQS based autoscaling. Respect downstream quotas: Limit embedding API requests/tokens and vector write rates; use bounded exponential backoff with jitter for transient failures. Make retries safe: Use stable document/version IDs, idempotent stage operations, durable checkpoints, dead letter handling, and an active version publication gate. Scale in conservatively: Drain leased work, finish or checkpoint processing, and avoid terminating workers mid write. 3. Beginner friendly numerical example Assume: Queue backlog: 500,000 documents New arrivals: 100 documents/minute Each worker completes 5 documents/minute on average Current fleet: 20 workers Current processing capacity = 20 × 5 = 100 documents/minute. That only matches new arrivals : the backlog does not shrink. If the target is to drain the backlog in 24 hours while still processing arrivals: Required backlog drain rate = 500,000 / (24 × 60) ≈ 347.2 docs/min. Required total processing = 347.2 + 100 ≈ 447.2 docs/min. Workers needed = ceil(447.2 / 5) = 90 workers. Important: Ninety workers is an illustrative capacity estimate, not an automatic scaling recommendation . It assumes steady document sizes, no retries, unlimited downstream capacity, and linear throughput. Benchmark whether OCR, embedding providers, and the vector database can sustain roughly 450 completed documents/minute first. If downstream embedding capacity permits only 300 docs/minute , then net drain capacity is 300 − 100 = 200 docs/minute , requiring 500,000 / 200 = 2,500 minutes ≈ 41.7 hours even with sufficient workers. The solution is then a quota/capacity or traffic priority decision, not more consumer processes. 4. A practical recovery plan 1. Protect customer facing search and critical revocation/update flows; isolate bulk imports from interactive services. 2. Confirm no data loss, poison message loops, or massively duplicated retries are inflating the backlog. 3. Establish measured maximum throughput for each stage and the end to end bottleneck. 4. Apply tenant quotas, bounded queues, and priority scheduling. 5. Gradually add workers while observing rate limits, OOMs, queue age, vector write latency, and successful search visibility. 6. Publish document versions only after durable indexing and completeness checks; surface per document progress to users. 7. Recalculate drain time periodically using observed completion rate, not theoretical peak worker capacity. 5. Trade offs and prevention More workers: Higher throughput until downstream quotas saturate; beyond that, retries and costs rise. Priority queues: Improve critical freshness but risk starvation; use aging or weighted fair scheduling. Batching: Cuts request overhead but increases per job memory and fault retry scope; bound batch size. Autoscaling: Adapts to spikes but needs worker startup time and conservative scale in. Admission control: Protects services but delays new uploads; communicate accurate status and expected service targets. Tenant isolation: Prevents noisy neighbor incidents but may leave spare capacity unused unless fairness rules can lend capacity safely. Prevention: Alert on oldest eligible document age, queue growth rate, estimated drain time, backlogs per stage/tenant, embedding quota utilization, workers at maximum capacity, retries, dead letter volume, and indexing freshness SLO breaches. Run a replay/load test with realistic PDF sizes, token counts, and multi tenant bursts. Interview takeaway: Queue depth alone is not a scaling plan. Calculate net drain rate = successful completion rate − arrival rate, identify the bottleneck, apply backpressure and priority fairness, and scale workers only within measured downstream capacity. References: AWS ECS — Scale services from SQS queue backlog per task: https://docs.aws.amazon.com/AmazonECS/latest/developerguide/service autoscaling queue.html AWS EC2 Auto Scaling — Backlog per instance and acceptable waiting time: https://docs.aws.amazon.com/autoscaling/ec2/userguide/as using sqs queue.html KEDA — ScaledObject autoscaling controls: https://keda.sh/docs/2.21/reference/scaledobject spec/

One malformed document repeatedly crashes a processing worker. How would you implement poison-message handling and a dead-letter queue?

Production incident (illustrative): A RAG service ingests customer PDFs from a queue. One malformed PDF crashes the parsing subprocess. The message becomes visible again, another worker picks it up, and the same crash repeats. Valid documents wait behind repeated failures, the queue age rises, and a dashboard reports a generic ingestion failed error. This is a poison message problem : an input repeatedly fails normal processing and needs controlled isolation. 1. Diagnose before retrying Identify the document: Correlate tenant id, document id, source version, message ID, processing stage, parser version, attempt count, and error fingerprint. Avoid logging confidential document contents. Find the failure boundary: Did the worker die at PDF validation, decompression, image rendering, OCR, chunking, or embedding? Inspect exit codes, OOM events, resource usage, stack traces, and container restarts. Check reproducibility: Replay the file in an isolated resource limited environment using an approved test copy. Determine whether it fails deterministically, or only during a temporary resource/API outage. Inspect the delivery mechanics: Compare queue receive counts, visibility timeout, retry delay, acknowledgment sequence, and whether one problematic document prevents other messages from progressing. Classify the error: Corrupt PDF, unsupported encryption, decompression bomb, invalid format, or consistently failing parser input is likely permanent without intervention. An embedding 429, a temporary object storage outage, or a brief database timeout is usually transient . 2. Implement poison message handling 1. Create a dead letter queue (DLQ): Route events that exceed an appropriate retry threshold from the primary ingestion queue to a restricted quarantine queue. In Amazon SQS, a redrive policy's maxReceiveCount controls this transfer; in RabbitMQ, dead letter exchanges and delivery limits provide analogous mechanisms. 2. Use bounded retries: Retry transient failures with exponential backoff and jitter; cap attempts and total processing time. For confirmed permanent errors, quarantine immediately using a reliable broker specific rejection or DLQ workflow rather than exhausting identical attempts. 3. Isolate untrusted parsers: Run PDF processing with CPU, RAM, runtime, page count, rendered pixel, and decompression limits. A crashing native parser should not take down the queue consumer or other jobs. 4. Preserve an investigation record: Store the safe source reference, checksum, document version, error class, stage, parser version, last attempt, and trace ID. Keep actual content in controlled object storage, not in broad access queue diagnostics. 5. Update document state accurately: Mark the file QUARANTINED / FAILED NEEDS REVIEW, not SEARCHABLE or silently SUCCESS. Let other documents proceed. 6. Alert and triage: Notify operators on unusual DLQ growth, repeated crash fingerprints, or high value tenant failures. Group matching signatures to avoid hundreds of duplicate incidents. 7. Replay after repair: Fix the parser, transform the input safely, or correct upstream data. Replay selected DLQ messages at a controlled rate, with deterministic document version IDs and idempotent upserts. Confirm the source version is still current before publishing. Broker nuance: DLQ movement and message acknowledgment semantics vary by platform. Set sufficient retention, verify destination accessibility, avoid a DLQ routing loop, and test whether a failed dead letter publication can lose a message. For FIFO workflows, routing a message away can affect ordering; explicitly address any per document sequencing guarantees. 3. Beginner friendly numerical example Suppose a batch contains 10,000 documents and 20 malformed PDFs . The queue retries every failing document 10 times , with each failed parse wasting 30 seconds of worker CPU time. Failed attempts: 20 × 10 = 200. Worker time wasted: 200 × 30 = 6,000 seconds = 100 worker minutes. With an error classifier and a policy that quarantines each reproducibly malformed document after 2 attempts , failures fall to 20 × 2 = 40 attempts. Wasted time becomes 40 × 30 = 1,200 seconds = 20 worker minutes. Illustrative saving: 100 − 20 = 80 worker minutes and less contention for the 9,980 valid documents. These are illustrative work totals, not wall clock delays; parallel workers and actual broker behavior change elapsed time. Do not use a universal two attempt threshold: configure limits by error type and infrastructure. 4. Practical recovery and operational checks Quarantine the identified document without deleting its source blob. Confirm the queue and worker fleet are healthy and good documents are draining. Inspect the file in a sandbox and determine whether the cause is malicious/malformed input, parser incompatibility, or a resource cap. Test a fixed parser or sanctioned conversion in staging using the same file signature. Replay one quarantined document version first, check chunk completeness, retrieval visibility, and that its status transitions to SEARCHABLE only after validation. Gradually replay the remaining matching failures with concurrency limits. Never blindly redrive a large DLQ into an unresolved defect. 5. Trade offs and prevention Low retry thresholds: Fast quarantine and low wasted work, but can wrongly sideline recoverable transient errors. High retry thresholds: More chances of recovery, but repeated crashes consume capacity and increase queue latency. Immediate quarantine: Good for deterministic parser failures, but requires reliable error classification. Parser sandboxing: Adds startup overhead, but protects shared ingestion infrastructure. Long DLQ retention: Helps investigation, but increases storage and privacy obligations; follow retention and deletion policies. Production signals: dlq message count, failures by stage and fingerprint, parser crash rate, attempt count, oldest unprocessed message, queue drain rate, quarantine age, replay success rate, and percentage of documents not searchable due to quarantine. Add fault injection tests with corrupt PDFs, oversize images, encrypted PDFs, and crash after write cases. Interview takeaway: Don't retry one unprocessable document forever. Classify failures, cap retry effort, sandbox parsing, quarantine poison messages with traceable metadata, and selectively replay after a verified fix —while allowing valid documents to continue. References: Amazon SQS — Using dead letter queues: https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs dead letter queues.html Amazon SQS — Capturing problematic messages: https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/capturing problematic messages.html RabbitMQ — Dead Letter Exchanges: https://www.rabbitmq.com/docs/dlx

Your system processes documents in multiple languages, but extraction quality suddenly drops for one language. How would you diagnose the regression?

Production incident (illustrative): A multilingual enterprise RAG system ingests English, Hindi, and Japanese policy documents. After a parser/OCR deployment, users report that Hindi documents are searchable by filename but return incorrect or incomplete answers. English remains healthy. The ingestion jobs still show SUCCESS because they produced some text. This is a language specific extraction quality regression . Diagnose extraction first, rather than immediately changing the embedding model, retriever, or LLM. 1. Confirm that extraction—not retrieval—is failing Take a small, authorized sample of affected Hindi documents and inspect the original page image, extracted text, and downstream chunks side by side . For the same documents, compare direct text extraction with OCR output. PDFs may have valid text layers, corrupt font encodings, invisible OCR layers, or scanned pages. Try an exact phrase search from a verified page and check whether the phrase exists in the extracted output before vector indexing. Separate extraction error , chunking/normalization error , embedding/retrieval error , and generation error using stage level artifacts. A retrieval failure alone does not prove OCR failure. Verify that the affected population has not shifted toward scans, poor resolution pages, or a new supplier/layout. 2. Root cause investigation: compare what changed 1. Segment the regression: Break metrics down by language, script, source connector, file format, scan/native text, page quality, tenant, OCR model version, parser version, and deployment cohort. A language label may be uncertain for short or mixed script pages. 2. Check language routing: Was Hindi (hi) accidentally routed to English only OCR? Was a forced language hint applied to mixed Hindi English pages? Did automatic detection start returning a different language or script? 3. Check models and dependencies: Compare OCR engine/model identifiers, deployed .traineddata files if using Tesseract, container images, fonts, locale settings, libraries, and provider API configurations between the last good and current releases. 4. Inspect layout and image preprocessing: Did the release change DPI, deskew, binarization, page rotation, cropping, or table extraction? Devanagari combining marks can be especially sensitive to character segmentation and image quality. 5. Inspect Unicode and cleaning: Check UTF 8 preservation, Unicode normalization choices, accidental ASCII only filtering, script detection, language specific punctuation, zero width characters, and whether cleaning unintentionally removes combining marks or legitimate characters. 6. Evaluate the extracted output: Look for empty pages, replacement characters (�), unexpected Latin only output, high repetition, missing glyphs, reduced character coverage, or altered reading order. Treat these as heuristics , not substitutes for ground truth. 7. Diff releases on identical documents: Process a fixed set through old and new pipelines with the same inputs. This distinguishes a software regression from a change in document mix. Provider nuance: Google Document AI supports OCR language hints and image quality analysis; Microsoft Document Intelligence can automatically detect many languages and warns that forcing a language code may produce incomplete output when the assumption is wrong. Validate supported scripts and features for the specific model and version in use. 3. Beginner friendly numerical example Consider 100 labeled Hindi pages , containing 20,000 reference characters in total. Compare OCR output with human verified transcription after applying the same explicitly defined normalization to reference and hypothesis. Old OCR release: 600 substitutions + 200 deletions + 200 insertions = 1,000 character errors . Old CER: (600 + 200 + 200) / 20,000 = 0.05 = 5%. New OCR release: 1,400 substitutions + 600 deletions + 400 insertions = 2,400 character errors . New CER: 2,400 / 20,000 = 0.12 = 12%. Regression: +7 percentage points , or 2.4× the previous CER. Now compare 100 similarly sampled English pages. Suppose CER remains at 3% for both releases. This strongly suggests a Hindi specific change , not a universal ingestion failure. All values are illustrative. Character Error Rate (CER) is defined as (substitutions + deletions + insertions) / reference characters; because insertions count, CER can exceed 100% in pathological cases. For real evaluation, report confidence intervals and error distribution by document/source as well as the aggregate. 4. Engineering recovery strategy 1. Contain further damage: Stop promoting suspect Hindi extractions; use the last validated pipeline where available. Quarantine affected new document versions rather than publishing low quality chunks. 2. Replay identical samples: Run old/current OCR and parser versions on the same documents; identify the earliest stage where outputs diverge. 3. Fix routing/configuration: Restore correct language model assets, supported OCR language detection or tested hints, image preprocessing, encoding, or script safe normalization. 4. Reprocess affected documents only: Use a durable source/version manifest and deterministic chunk IDs. Re extract, re chunk, regenerate embeddings where text changed, and stage them under a new index generation. 5. Verify before activation: Require per language extraction gates, representative spot checks, expected page/chunk coverage, and authorized search tests. Keep a validated previous generation available for rollback within retention policy. 6. Track customer impact: Identify the document versions, downstream answers, citations, and tenant cohorts that used defective extracted text. Reindex and retest high priority sources first. 5. Prevent the next language specific incident Maintain a golden dataset for each major language and script, including native PDFs, scans, forms, mixed script pages, rare characters, tables, and multiple image qualities. Track CER/WER when labeled transcriptions exist, plus page text coverage, OCR failure ratio, detected script agreement, language routing distribution, and successful source to search publication. Report per language , not just global averages. Test OCR/parser upgrades with shadow runs and language specific canaries before rolling out broadly. Set regression thresholds per language, document type, and business use case. Example: block deployment if Hindi CER increases by more than 2 percentage points on a representative benchmark; choose the actual threshold based on baseline and impact. Log source, model, language hint, parser, preprocessing, and normalization versions for reproducibility, without exposing private page content. Use selective manual review for critical legal, healthcare, or financial documents. Trade offs: Per language OCR routing and quality gates cost additional compute and maintenance. Automatic language detection is easier for mixed content but can misroute short/noisy pages; forced hints help when metadata is reliable but can damage mixed language documents. Aggressive Unicode cleanup can improve consistency while silently deleting meaningful script characters. Evaluate each choice on representative source documents. Interview takeaway: When one language degrades, stratify the failure, compare identical documents across releases, inspect routing/OCR/Unicode processing, measure language specific CER, roll back the defective stage, and reindex only affected versions . A green ingestion status is not proof of readable text. References: Google Cloud Document AI Enterprise OCR: https://docs.cloud.google.com/document ai/docs/enterprise document ocr Microsoft Azure Document Intelligence language support: https://learn.microsoft.com/en us/azure/ai services/document intelligence/language support/ocr?view=doc intel 4.0.0 Microsoft multilingual Tesseract OCR configuration: https://learn.microsoft.com/en us/power automate/desktop flows/how to/ocr multilingual documents

A user uploads a document but cannot retrieve it immediately. How would you explain and manage read-after-write consistency?

Production incident (illustrative): A user uploads a new HR policy PDF at 10:00. The API responds 201 Created, but the user immediately asks the RAG assistant about it and gets “No relevant documents found.” At 10:01 the same question works. The upload succeeded; the searchable index had not caught up . This is a read after write consistency and user expectation problem. 1. Explain the consistency boundary A RAG ingestion system performs several distinct operations: Upload accepted: Original bytes are durably stored; this says nothing about the document's search visibility. Processing: Parsing, OCR, chunking, embeddings, and indexing are asynchronous and can take seconds or minutes. Write acknowledged: The vector or search backend accepts the write; depending on the engine, a refresh, replica propagation, or asynchronous indexing may still be pending. Searchable: A query on the actual active retrieval path , in the correct tenant/collection and under the user's permissions, can find the intended document version. Read after write consistency means a client that completed a write can subsequently observe that write under the documented read interface. Ordinary asynchronous RAG ingestion often offers eventual search visibility , not immediate read after write search consistency. Do not equate upload succeeded with search ready. Elasticsearch documents that index writes are near real time for search: refresh=wait for waits for a refresh to expose a write, while forcing refresh=true can hurt indexing throughput. These options govern Elasticsearch's index refresh; they do not make the upstream OCR/embedding pipeline synchronous. Weaviate documents tunable consistency and asynchronous indexing/replication considerations, so guarantees must be checked for the actual backend. 2. Diagnose where the delay occurs 1. Correlate upload id, tenant id, document id, source version, ingestion run id, and index generation. 2. Record timestamps for upload stored, queue accepted, worker started, parsed, embedded, vector write acknowledged, index refreshed/visible, and first successful authorized retrieval. 3. Compare exact document ID lookup with search using the correct tenant, active index alias, access filters, and source version. A document may exist in storage but not the searchable index. 4. Test whether the problem is indexing lag (queued work), refresh/replica lag (written but not visible), wrong index/namespace , permissions , or ranking (present but not in top k). 5. Check p50/p95/p99 upload to search latency rather than relying on an average. Investigate outliers caused by scanned PDFs, rate limits, or worker backlog. 6. Retry visibility probes with a bounded deadline and backoff. Avoid assuming that one unsuccessful search means permanent data loss. 3. Beginner friendly numerical example Suppose one upload completes these stages: 10:00:00 — file saved; API returns 202 Accepted with document id=doc 42. 10:00:04 — worker begins parsing. 10:00:12 — parsing and chunking complete. 10:00:20 — embeddings complete. 10:00:23 — vector upserts acknowledged. 10:00:27 — expected document/version verified through the active authorized search path. Then: Upload to searchable latency: 27 seconds. Time after vector write acknowledgment but before visibility: 4 seconds. If the product promises 95% of eligible documents searchable within 60 seconds , this upload passes. One passing upload does not prove the overall 95% SLO; calculate the percentage across a representative time window and document types. These numbers are illustrative. “Visible in exact ID search” and “ranked highly for every natural language query” are separate properties. 4. Engineering solution: make readiness explicit 1. Expose document states , such as UPLOADED → QUEUED → PROCESSING → INDEX WRITTEN → SEARCHABLE (or FAILED/QUARANTINED). Record transitions durably and include document version. 2. Return the right API response: 202 Accepted plus a document ID and a status endpoint for asynchronous uploads. 201 Created may accurately describe resource creation, but the UI must not imply search readiness. 3. Provide status polling or events: Offer GET /documents/{id}/status, webhooks, or server sent events so the client knows when search is ready. Use exponential polling backoff rather than querying continuously. 4. Gate SEARCHABLE on verification: Check the manifest's expected chunk count, correct generation and tenant, permitted read scope, and provider appropriate search visibility. An upsert acknowledgment is insufficient. 5. Offer session specific read your writes only when justified: For critical UX, temporarily route the uploader's questions to a securely authorized temporary document store or recently ingested chunks, clearly marked as processing. Ensure the same ACL checks, expiration, citation provenance, and bounded resource use; never bypass permissions or claim the document is in the global index. 6. Use backend appropriate visibility controls: For Elasticsearch/OpenSearch, consider batched refresh=wait for when a synchronous search after index contract is needed; avoid forcing a refresh after every bulk write. For vector databases, follow documented consistency controls or verify reads through the active query path. 7. Handle permanent failures: If deadline expires, mark DELAYED or FAILED, retain the original upload, explain the state to the user, and use idempotent retries/checkpoint recovery. Do not silently return a final “not found” for a known in progress upload. 5. Trade offs and prevention Synchronous wait: Simpler immediate UX but longer API response times, timeouts, and lower throughput for large documents. Asynchronous readiness: Scales better and avoids blocking uploads, but needs a clear progress/status experience. Forced refresh or stronger consistency: Reduces visibility delay in supported engines but can increase I/O and latency. It does not fix slow parsing or missing embeddings. Temporary session overlay: Can give the uploading user earlier answers, but introduces extra authorization, freshness, and consistency complexity. Monitor upload to searchable latency by file type/tenant, queue wait time, embedding latency, index refresh lag, stuck PROCESSING documents, the percentage meeting the freshness SLO, and successful authorized read after write canary checks. Test replicas, index aliases, permission changes, and delayed OCR under production like load. Interview takeaway: A successful upload is not the same as a successful searchable write. Make the consistency contract explicit, trace every stage, expose readiness states, measure upload to search latency, and use search visibility confirmation or a carefully authorized read your writes path when the UX requires it. Research references: Elasticsearch refresh parameter (official): https://www.elastic.co/docs/reference/elasticsearch/rest apis/refresh parameter Elasticsearch index refresh API (official): https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation indices refresh Weaviate consistency and replication (official): https://docs.weaviate.io/weaviate/concepts/replication architecture/consistency

Your assistant still references a deleted document several hours after deletion. How would you trace every remaining copy?

Production incident (illustrative): At 09:00, a customer deletes an internal salary policy PDF from SharePoint. At 13:00, the RAG assistant still quotes a sentence from that PDF and cites its old URL. The source is gone, but deletion from one system does not automatically delete every derived copy . Treat this as a possible information governance and authorization incident until verified. 1. Contain first: stop serving deleted material 1. Identify the canonical tenant id + source id + document id, all source versions, and the deletion timestamp. 2. Immediately write a durable deletion tombstone or REVOKED/DELETING state in the authoritative document registry. The retrieval gateway must reject that document ID before any content reaches the LLM , even if the search index is stale. 3. Invalidate related retrieval, snippet, answer, and conversation context caches as supported. Stop in flight ingestion jobs from republishing the source by checking the tombstone at activation. 4. Audit whether the user received a newly retrieved chunk, a cached answer, or text already stored in a prior conversation. These are different mechanisms and remediation paths. 5. Escalate actual unauthorized disclosure under the organization's privacy/security incident process; minimize raw sensitive content in diagnostics. Important: A model can sometimes repeat a claim from its own pretrained knowledge or the conversation history without retrieving a live document. A textual match is evidence to investigate, not proof the vector store still holds the file . 2. Trace the document's complete lineage Start with document id, source URI, canonical content hash, version IDs, tenant, and ingestion run ID. Enumerate every derived artifact: Source and intake: SharePoint item, webhook/delta event, uploaded object storage copy, connector staging folder, upload manifests, and retry queues. Extraction: temporary files, OCR images, extracted text, layouts, tables, page level JSON, and parser checkpoint artifacts. Chunk and embedding layers: chunk catalog, embedding cache, vector index namespaces/shards, and previous index generations. Search by stable parent document id and all versions , not just filename or content text. Other retrieval stores: BM25/keyword indexes, full text databases, graph nodes/edges, reranker candidate caches, replicas, snapshots, and alternate collections. Answer assembly: retrieval caches, prompt/context caches, generated answer caches, citation catalogs, tool result caches, session state, stored conversations, and materialized summaries. Operational copies: dead letter queues, event logs, telemetry payloads, backups, export buckets, disaster recovery replicas, and third party providers; follow their actual retention/deletion policies rather than claiming every copy is instantly erasable. Record for each artifact its owner, location, lineage key, deletion operation, acknowledgment, verification timestamp, and retention exception . 3. Debug the deletion pipeline 1. Did the source deletion generate an event? Inspect connector delta/watermark progress, webhook delivery, polling gaps, and soft delete to hard delete transitions. Some indexers need deletion markers before source content disappears; otherwise orphaned search records can survive. 2. Did the deletion worker fan out correctly? Check event ledger, queue retries/dead letter state, target list, batch errors, and access to each namespace/index. 3. Did indexing actually remove searchable chunks? Compare the expected chunk ID manifest with exact ID retrieval and filtered search. Index delete acknowledgment can precede physical storage reclamation. 4. Is another index still active? Check alias routing, replicas, blue/green index generations, tenant partitions, and delayed rebuild snapshots. 5. Did caches survive? Reproduce with caches disabled; then inspect retrieval/response cache keys and invalidation versions. A previous conversation may retain the literal excerpt even after retrieval is fixed. 6. Did an old worker resurrect the document? Look for indexing tasks started before deletion and completed afterward; enforce tombstone/version checks before write activation. 7. Could a similar current source explain the answer? Compare cited document id, hash, retrieved chunk IDs, and trace logs rather than guessing from wording. Microsoft's Azure AI Search documentation explicitly notes that indexers can leave orphan documents if source deletion occurs before their deletion detection step. Elasticsearch describes near real time search refresh and asynchronous bulk deletion details. These are backend specific examples , not universal timing guarantees. 4. Beginner friendly numerical example Suppose one deleted document generated: 80 vector chunks in the primary index 80 entries in a blue/green standby index 80 keyword search entries 6 cached retrieval results and 4 cached answers That's 80 + 80 + 80 + 6 + 4 = 250 indexed or cached artifacts , assuming each listed artifact is distinct. The deletion worker removes the 80 primary vectors, but the standby index and keyword index retain 80 + 80 = 160 searchable entries, and the caches retain 10 copies. Indexed entry cleanup after the first step: 80 / 240 = 33.3%. 160 index entries still require deletion or an enforceable global tombstone. 10 cache entries must also be invalidated. This does not mean 250 independent plaintext copies (vectors are numeric representations), and artifact counts do not measure whether content is still retrievable. Verify the actual read path. All figures are illustrative. 5. Engineering solution: deletion as a tracked workflow 1. Set a high priority, durable tombstone (tenant, document, delete generation); block query time use immediately. 2. Use an outbox or durable event ledger to schedule cleanup in every registered derivative store. Correlate work by a single deletion request id. 3. Drive cleanup from a document to artifact manifest ; delete all versions and stale index generations, not just current chunks. Use idempotent deletes and bounded retries. 4. Version or invalidate every authorized cache that can contain the source, including answer/citation caches. For existing conversation history, apply the product's explicit retention and deletion rules; do not silently rewrite past user conversations. 5. Fence concurrent indexing: workers must check the tombstone and source version before publishing. Deletion must dominate obsolete ingestion events. 6. Verify each active retrieval implementation using exact ID lookups, keyword/vector searches, tenant scoped citation checks, and previously affected user queries; verify after expected index refresh lag. 7. Reconcile source registry versus all active indexes on a schedule. Keep a minimal audit record of the deletion operation without retaining source text unless a documented obligation requires it. 8. Handle backups and third party copies explicitly: document retention intervals, access controls, legal holds, restore time deletion replay, and any provider limitations. Do not promise instant physical erasure if immutable backups remain under policy. Simplified safe workflow The cross system operation is not one atomic transaction . If a store is unavailable, keep the tombstone enforced while retries and reconciliation complete. 6. Trade offs and prevention Immediate query time tombstones: Fast containment, but add a lightweight policy lookup or replicated deny set. Fail closed if the policy service is unavailable for sensitive content. Physical deletion: Necessary for retention/data erasure obligations, but may be asynchronous and provider specific; deleted storage bytes are not the same thing as blocked retrieval. Comprehensive lineage manifests: Make deletion auditable but require maintaining mappings across transformations, versions, and stores. Cache TTL alone: Easy to implement but leaves a temporary exposure window. Use event invalidation plus bounded TTL as defense in depth. Backups: Protect disaster recovery but may be immutable. Restrict restoration and replay deletion logs before resuming search; disclose retention constraints. Monitor: deletion request to query block latency, oldest outstanding deletion, incomplete targets, orphan chunk count, stale cache hit attempts, index generation mismatches, tombstone bypass attempts, and restore time replay success. Regularly create and delete canary documents to assert that no active authorized search path still serves them. Interview takeaway: Deleting a source file is only the beginning. Immediately deny retrieval, trace the document through every derived store and cache, delete or quarantine all indexed versions, fence old workers, verify user visible absence, and document backup/retention limits. Research references: Azure AI Search — Delete documents and orphaned records: https://learn.microsoft.com/en us/azure/search/search how to delete documents Elasticsearch — Delete by query: https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation delete by query Elasticsearch — Near real time search: https://www.elastic.co/docs/manage data/data store/near real time search

A company has five versions of a policy, and the RAG system regularly retrieves outdated versions. How would you enforce effective-date-aware retrieval?

Production incident (illustrative): An enterprise RAG assistant answers “How many days of parental leave do employees get?” using a five year old policy that ranks highly because its text closely matches the question. The knowledge base contains five versions of the policy, but the retriever treats all five as equally eligible. An answer can be textually grounded yet wrong for the applicable date . This is an effective date and policy authority problem , not just a stale embedding problem. The policy with the newest upload timestamp is not necessarily the one that was legally effective when the employee needed guidance. 1. Clarify which date the question is asking about Current policy: “What is our parental leave policy today?” Resolve “today” in the organization's agreed timezone and use that date as the policy's as of date . Historical policy: “What was the policy on 15 March 2023?” Use 15 March 2023 , not today's date. Future dated policy: “What changes next January?” Retrieve planned/future effective policies only if authorized and explicitly describe their status; a scheduled revision is not necessarily approved. Ambiguous request: Ask for the relevant date or clearly state the date assumption. Some policies also depend on country, employment type, department, or legal entity . Do not decide applicability using date alone. Separate three timestamps: source publication/approval date , valid from/valid to (business effective dates) , and ingestion/indexing time . The newest document in the vector store may contain a future policy or a retrospective correction. 2. Debug why outdated policies appear 1. Reproduce the failing question and capture its as of date, tenant, jurisdiction, retrieval filters, top candidates, document IDs, policy IDs, scores, and citation versions. 2. Inspect the five policy records: are valid from and valid to present, correctly parsed, and inherited by every chunk ? Are null end dates treated consistently? 3. Check whether the retriever uses a hard eligibility filter before semantic ranking , or merely adds a small freshness bonus. A relevance score can otherwise rank an expired policy first. 4. Verify that draft, withdrawn, superseded, and future effective versions are ineligible for current policy answers, according to the source of authority. 5. Check both effective time (when the rule applied) and recorded time (when the organization learned or recorded the version). A correction entered later may change how a historical query must be interpreted. 6. Examine caches, index aliases, and context assembly. Even with a correctly filtered search, an unfiltered keyword retriever or cached passage can reintroduce an obsolete version. 7. Confirm that the source documents do not overlap in effective dates for the same policy scope without a documented precedence rule. 3. Model policy versions and authority explicitly For each policy version, persist: Stable policy id, version id, source system ID, content hash, chunk IDs, and original citation URL/page. valid from (inclusive) and valid to (exclusive), with an unbounded end only when genuinely open ended. approval status (e.g., approved, draft, withdrawn), supersedes version id, owning department, and authoritative source priority. Tenant, jurisdiction, employee segment, and ACL metadata. recorded at or change history fields if audit or historical reconstruction requires bitemporal behavior. An interval such as [2025 01 01, 2026 01 01) applies from January 1, 2025, until just before January 1, 2026; there is no ambiguity at midnight on the boundary. PostgreSQL daterange supports interval containment and overlap checks; search engines such as Elasticsearch provide range filters. These building blocks do not decide legal precedence for you—business ownership does. 4. Beginner friendly numerical example Assume one approved company policy with five distinct versions: v1: 2021 01 01 to 2022 01 01 — 10 days v2: 2022 01 01 to 2023 01 01 — 12 days v3: 2023 01 01 to 2024 07 01 — 14 days v4: 2024 07 01 to 2026 01 01 — 16 days v5: 2026 01 01 onward — 18 days All intervals are start inclusive and end exclusive. For a current query as of 5 November 2026 , only v5 is effective, so the answer is 18 days (subject to the user's scope and the source policy's authority). For 15 March 2023 , select v3 , giving 14 days . Suppose vector similarity ranks the passages v2: 0.93, v5: 0.85, and v3: 0.82. Do not choose v2 just because it scores highest. Filter by effective date and applicability before similarity ranking. The numerical policy values and scores are illustrative, not legal or HR advice. If each version creates 40 chunks, the index contains 200 chunks . With date aware filtering, the current query searches only the 40 eligible v5 chunks instead of 200 candidates across conflicting versions. This improves temporal correctness and can reduce retrieval work, depending on the backend. 5. Engineering design: policy aware retrieval 1. Ingest each policy as a separate immutable version ; do not delete historical versions needed for legitimate as of queries. 2. Parse dates and approval status from an authoritative policy registry or verified metadata , not from an LLM's guess about a PDF title. Resolve missing or conflicting validity fields through an owner review queue. 3. Validate intervals at publish time: valid from < valid to when there is an end date; detect overlaps/gaps for the same policy, jurisdiction, and population. Explicitly model exceptions and approved precedence. 4. Resolve as of date and policy scope from the query/session, applying timezone rules. Historical queries must preserve the requested date. 5. Apply mandatory metadata/authorization predicates before retrieval: tenant id = current tenant approved = true valid from <= as of date valid to IS NULL OR as of date < valid to jurisdiction / employee type / ACL match the caller and question. 6. Run keyword/vector retrieval and reranking within the eligible set , not across all versions. For a query comparing multiple years, explicitly retrieve each requested interval separately. 7. Validate that all evidence used in one current policy answer refers to compatible policies and effective intervals; cite version and valid dates , not just the generic policy title. 8. Invalidate caches when an owner approves, withdraws, or corrects a policy; include as of date, scope, and policy registry version in cache keys where appropriate. 9. If no unambiguous authoritative version is eligible, abstain or escalate instead of silently returning the closest outdated policy. Implementation sketch The selected version id or authorized eligible IDs then constrain vector/hybrid search. In PostgreSQL a daterange(valid from, valid to, '[)') @ :as of date::date predicate is an alternative. Use exclusion constraints or validation logic to flag inappropriate overlaps for the same policy scope. 6. Recovery and prevention Contain affected answers: If a wrong effective policy has been served, identify involved answer traces and policy IDs, prioritize critical workflows, and correct the registry/filter logic. Backfill missing dates: Audit the full five version chain, approvals, jurisdictions, and supersession links with policy owners. Reindex only when needed: If date metadata can be patched independently, avoid unnecessary re embedding; reprocess when chunk content or source provenance has changed. Run temporal test cases: Test the exact day before a policy changes, the first effective day, historical queries, future policies, revoked policies, and different employee groups. Monitor: percentage of answers using ineligible versions, missing effective dates, overlapping intervals, cache/version mismatches, as of query failures, and source registry freshness. Maintain auditability: Record resolved as of date, version IDs, policy scope, approval state, prompt/model/index revision, and citations so a disputed answer can be reconstructed. Trade offs: Hard validity filters improve accuracy but can hide documents when metadata is incomplete; fixing metadata is safer than weakening policy eligibility. Retaining historical versions enables legitimate time travel queries but adds storage and governance requirements. Live authorization and registry checks offer fresher policy decisions at the cost of latency and operational dependency. Interview takeaway: “Latest uploaded” is not “effective on the requested date.” Maintain authoritative version intervals, resolve an as of date, apply hard date/approval/access filters before ranking, cite the selected version, and abstain when applicability is unresolved. References: PostgreSQL range types and validity intervals: https://www.postgresql.org/docs/current/rangetypes.html Elasticsearch range queries: https://www.elastic.co/docs/reference/query languages/query dsl/query dsl range query Elasticsearch boolean filter clauses: https://www.elastic.co/docs/reference/query languages/query dsl/query dsl bool query

A document moves between folders without changing its content. How would you preserve its identity, citations, and access permissions?

Production incident (illustrative): An employee moves a financial forecast from /Shared/General/ to /Finance/Restricted/ in SharePoint without editing its contents. The RAG connector treats paths as document identities, so it indexes a “new” document under the restricted path while leaving the old vectors and citations under the open path. Worse, the old chunks still use the General folder's permissions. A search may now reveal confidential information to people who no longer have access. This is a move detection, stable identity, citation resolution, and authorization incident—not a reason to generate new embeddings automatically. 1. Diagnose what changed Compare the source document's stable item ID, drive/site ID, parent folder ID, old/new paths, content hash, source version, modification time, and permission metadata. Review Microsoft Graph delta responses, connector cursors, and move/rename notifications. Microsoft's delta guidance recommends tracking items by ID rather than path , since a folder rename does not necessarily produce individual changes for every descendant. Query both old and new RAG paths or IDs, including keyword indexes, vector shards, older index generations, and caches. Determine whether the connector created duplicate logical documents or left orphaned chunks. Follow one known citation to its target. Is it a stale path based URL, a durable ID backed resolver, or an old cached link? Compare effective permissions after the move with index ACLs. Moving a file between folders can change inherited permissions; don't assume a content preserving move preserves authorization. Identify the move type. A normal within drive move can be represented by updating a DriveItem's parentReference; a cross drive or cross site transfer may create a new item ID and require explicit identity reconciliation. Do not assume IDs survive all migrations. 2. Immediate containment 1. Mark the affected document's authorization state REVALIDATING; block or re check access until its new effective ACL is trusted. 2. Invalidate cached search results, excerpts, generated answers, and citation destinations that refer to the old location. 3. Stop workers handling stale events from republishing old folder metadata or old access grants. 4. Inspect retrieval/audit records for possible unauthorized disclosure. Follow the organization's incident process if an exposure occurred. 5. Restore search only after authorized read tests succeed under the new location. 3. Engineering design: identity independent of location Assign the RAG record a stable internal canonical document id; link it to a source drive id + item id when that combination is stable in the source's scope. Store path, folder/parent ID, display name, and source URL as mutable location attributes , not as the primary identity or vector ID. Maintain a source identity map or lineage ledger linking canonical IDs to source IDs, historical locations, moves, and any provider issued replacement IDs. Preserve content derived chunk IDs, version, and embeddings when the content hash and processing configuration are unchanged . A location only update normally needs metadata and ACL reconciliation, not OCR and embedding regeneration. Preserve citations as canonical document id + source version + page or chunk anchor. Resolve a current permitted destination URL at answer time, or show an explicit stale link/permission error. Avoid guessing URLs from old folder paths. Fetch or recompute effective source ACLs after the move, including inheritance, direct sharing, and group memberships, then update the authorization layer. Enforce permissions before retrieved passages reach the LLM . Version metadata/ACL updates and use a monotonic change token or compare and swap to prevent a delayed old folder event from restoring outdated paths or permissions. Reconcile old index entries and caches using the canonical ID. When a cross drive move changes source IDs, confirm provenance before linking; a content hash match alone can confuse distinct identical files. 4. Beginner friendly numerical example Assume a 100 page document produces 80 chunks . It moves from a folder visible to 200 employees into a restricted folder visible to 15 employees , while its bytes and extracted text remain identical. A path keyed ingestion pipeline incorrectly treats the move as a new file: Old path: 80 vectors , still visible to 200 employees. New path: another 80 vectors , visible to 15 employees. Total: 160 stored vector records , including 80 redundant records . 185 employees who were formerly eligible (200 − 15) may still be authorized by the stale old path index , even though they should no longer see the document. These counts are hypothetical; actual permission overlap and access depend on the source's effective ACLs, and vector duplication does not by itself prove a leak. With a stable canonical ID, the expected result is still 80 logical chunks , unchanged embedding payloads, a new location, and updated authorization . Confirm that the 185 now ineligible users cannot retrieve the document through any active search or cache path. 5. Step by step safe handling 1. Consume source delta/change notifications and resolve the latest item by its stable provider ID; record the move's old and new parent IDs. 2. Compare content hash and processing version. If unchanged, skip expensive parsing, chunking, and re embedding. 3. Mark authorization revalidation pending, query authoritative effective permissions, and update the secure ACL reference or index metadata. 4. Update the canonical document's current path and citation resolver, with a versioned, atomic metadata transition where possible. 5. Query by canonical ID to identify duplicate/stale index records; remove or deactivate old records using idempotent operations. 6. Verify search from representative allowed and denied user identities, check citations, and test stale caches/replicas. 7. Commit the new source checkpoint only when move handling is durable, and record an audit trail of locations and ACL changes. Simplified pseudocode This is architecture pseudocode. Source change token formats, ordering, and atomic update support are connector specific. 6. Trade offs and prevention Path as ID: Easy to implement but unreliable on rename/move; causes duplicates and stale links. Stable provider ID: Better for within source moves, but may change during cross drive migrations or copy/delete workflows. Add a canonical identity mapping and explicit migration matching. Content hash reuse: Saves compute, but never use hash equality as evidence that permissions, location, ownership, or identity are unchanged. Live ACL checks: Improve revocation safety but add latency and source dependency; use deny on uncertain policies for sensitive documents. Citation redirects: Improve continuity, but never redirect unauthorized users to a previously accessible path. Monitor: moves treated as creates/deletes, duplicate chunks per canonical document, unresolved source IDs, stale citation URLs, ACL reconciliation lag, unauthorized retrieval canaries, old path cache hits, and cross drive migration failures. Include tests for folder renames, moves into/out of restricted folders, bulk moves, direct permissions, and out of order delta events. Interview takeaway: Document identity, location, content, and permissions are four different things. A folder move may change location and ACLs without changing content. Preserve the canonical document/chunk identity, update citations and authorization from the authoritative source, remove stale paths, and verify the new access boundary before serving the document. References: Microsoft Graph — Move a driveItem: https://learn.microsoft.com/en us/graph/api/driveitem move?view=graph rest 1.0 Microsoft Graph — DriveItem delta: https://learn.microsoft.com/en us/graph/api/driveitem delta?view=graph rest 1.0

A source system changes document IDs during migration. How would you maintain source-to-chunk lineage?

Production incident (illustrative): A company migrates 50,000 documents from a legacy SharePoint site into a new document platform. Old source IDs such as SP 01847 are replaced by destination IDs such as DOC 99302. The RAG pipeline interprets these as new documents, creates duplicate embeddings, breaks stored citations, and loses the ability to prove which source produced a given chunk. Some old ACL metadata also survives. This is a cross system identity and lineage migration problem . It differs from a simple folder move: the original provider issued ID may genuinely disappear, while the underlying business document is logically the same. 1. Diagnose the failure and define identity Select affected answers and trace citation → chunk id → document version → canonical document id → current source reference. Find the first missing join. Compare pre/post migration manifests: old and new source IDs, source system/tenant namespaces, paths, version identifiers, content hashes, sizes, modification timestamps, ACLs, and chunk counts. Check if the migration was a copy, move, transformation, conversion, merge, split, or delete/recreate. A filename or hash match alone is not enough to establish identity. Inspect source change feeds and migration tool audit logs for old to new mappings. Microsoft Graph delta advises tracking within drive items by ID instead of path; this does not mean IDs are portable across source migrations. Search both indexes for duplicate chunks, orphan chunk records, stale citations, and versions promoted from the wrong source. Verify that the search gateway still applies current destination permissions before returning any evidence. 2. Build a durable identity crosswalk Assign an internal canonical document id that does not depend on either vendor's IDs. Persist an auditable mapping: canonical document id: stable identity used by the RAG system. source system, source namespace / tenant, source document id, and source version. destination system, destination namespace, destination document id, and destination version. Original and new URLs/paths, content checksum, migration batch/run ID, provenance, matching method, match confidence, and validation status. ACL mapping, ownership, and a superseded at or cutover timestamp. chunk id, page/section anchor, chunker version, embedding model version, and index generation in a separate canonical document → source version → chunks lineage manifest. Treat provider IDs as opaque and scoped: (source system, tenant, container, provider id) may be needed to make an identity unambiguous. Avoid unsafe automatic matching Prefer a migration export that explicitly pairs source and destination IDs. For incomplete mappings, use content hashes, file properties, metadata, and structural evidence as candidate signals , then validate business ownership. Same hash does not prove same document: Two departments may upload identical blank templates with different access permissions. Different hash does not prove a new document: Office to PDF conversion, line ending changes, or metadata edits can change bytes while preserving logical identity. Merges and splits need many to many lineage: Never force two source documents into a one to one mapping without a documented transformation. Quarantine ambiguous matches for review instead of exposing them under guessed permissions or citations. 3. Engineering migration workflow 1. Snapshot a source manifest including IDs, versions, checksums, source ACLs, and all indexed chunk IDs before migration begins. 2. Ingest migration mappings first , using uniqueness constraints on each scoped provider identity and on approved one to one mappings; retain transformation specific lineage for merges/splits. 3. Build or update a canonical registry and versioned mapping table. Keep changes append only/auditable to support rollback and historical investigations. 4. Revalidate access control using destination effective ACLs and the current caller identity. Fail closed for records with unresolved permissions. 5. Re use embeddings only when safe: If the extracted content, chunking algorithm, chunk boundaries, and embedding model are identical, map existing chunks to the canonical document. Otherwise reparse/rechunk/re embed the affected versions. 6. Stage the destination index generation without deleting the old active index. Keep citations as stable canonical document id + version + page/section/chunk anchor, then resolve the destination URL at read time. 7. Verify the migration with manifest counts, random and risk weighted source to chunk lineage tests, exact ID retrieval, authorized/denied access checks, and sample citations that open the intended destination file. 8. Switch routing atomically where supported (index alias or authoritative registry pointer). Use version fencing so late old source events cannot reactivate obsolete IDs. 9. Retire old records after a hold period once search, citation, deletion, and permission checks pass. Preserve lineage audit records according to the organization's retention policy. OpenLineage provides a general model for tracking relationships among jobs, runs, and datasets. Its events/facets can complement this design, but a document/chunk crosswalk is application specific and must be implemented separately. 4. Beginner friendly numerical example Suppose the legacy system has 10,000 documents , each producing 20 chunks , so the index contains: 10,000 × 20 = 200,000 chunks. The migration tool exports confirmed old to new ID mappings for 9,700 documents . The remaining 300 documents need reconciliation: 9,700 / 10,000 = 97% confirmed identity coverage. If the new connector treats all 10,000 destination IDs as independent documents and reindexes them, it may add another 200,000 vectors , resulting in 400,000 vector records before cleanup. With canonical mapping and validated unchanged chunks, the 9,700 mapped documents can retain 194,000 logical chunk identities (9,700 × 20). The other 6,000 chunks (300 × 20) must be withheld from automatic identity reuse until documents are matched, approved as genuinely new, or reprocessed. The numerical example is illustrative; physical index storage may temporarily contain dual generations during a safe cutover. Distinguish temporary migration copies from true long lived duplicate logical documents. 5. Root cause recovery for already broken citations Find a failing citation's prior chunk id and resolve its stored canonical ID or legacy source ID against the migration crosswalk. Verify that the mapped destination version has corresponding content and page/section anchor; page numbers can change after conversion. Update the citation resolver, rather than rewriting historic answers with unverified new links. If the anchor cannot be established, surface an explicit unavailable/needs review citation. Rebuild only affected chunk metadata and index records. Avoid needless re embedding when vector payloads remain valid. Invalidate stale retrieval/answer caches and check historical audit records without retaining or exposing unauthorized content. Run a reverse lineage test : every currently searchable chunk must resolve to exactly one intended canonical version or a clearly modeled many to many transformation. 6. Trade offs and prevention Canonical ID layer: Adds registry and mapping complexity, but decouples RAG identity from vendors, paths, and migrations. Hash based reuse: Saves compute but risks conflating different documents; require namespace, ownership, version, and explicit matching evidence. Blue/green migration: Costs extra storage temporarily but makes verification and rollback much safer. Strict unresolved item quarantine: Delays availability, but prevents wrong citations and ACL leakage. Append only lineage: Improves audits but needs controlled retention, redaction, and access policies. Monitor unmapped source IDs, mapping conflicts, orphan chunks, duplicate canonical IDs, failed citation resolutions, ACL mismatches, migration drift, and lineage coverage percentage . Test files with duplicate contents, renamed files, converted formats, deleted sources, split/merged documents, and destination permission changes. Interview takeaway: Provider document IDs are not stable business identity across migrations. Maintain a canonical document registry and verified source ID crosswalk, preserve version to chunk provenance, stage and validate index cutover, and never use content equality as a shortcut around authorization. References: Microsoft Graph driveItem delta: https://learn.microsoft.com/en us/graph/api/driveitem delta?view=graph rest 1.0 OpenLineage core concepts and facets: https://openlineage.io/docs/spec/facets/job facets/ Amazon S3 object version identifiers: https://docs.aws.amazon.com/AmazonS3/latest/userguide/versioning workflows.html