How would you design an auditable document-state machine that tracks ingestion, indexing, publication, supersession, and deletion?

Production incident (illustrative): An enterprise RAG assistant quotes a confidential policy PDF three hours after the source owner deleted it. Logs show the ingestion job finished, a vector delete timed out, and a second worker retried an old publish message. The monitoring dashboard says the document is "processed," but nobody can prove which version was visible at any moment. The incident exposes a governance design flaw: background tasks are not a document lifecycle, and a generic "success" status is not evidence that publication, supersession, or deletion actually completed.

1. Contain and trace the incident

• Immediately block the affected tenant_id + canonical_document_id at the retrieval authorization/visibility gate. Invalidate answer, retrieval, citation and CDN caches; halt late publish jobs for that document.
• Preserve the source deletion notice, worker run IDs, event identifiers, vector deletion receipts, active-version pointer history and user-visible citations.
• Build a timestamped timeline: source_deleted_at → deletion_requested → serving_blocked → vector_delete_requested → vector_delete_verified → purge_complete.
• Determine whether the system acknowledged a queue message before persisting its state transition; whether a stale worker republished a tombstoned version; and whether partial cleanup was mistaken for completion.
• Root cause: mutable status fields and disconnected worker logs had no guarded transitions, durable event audit, or generation fencing. An asynchronous delete and a delayed publish could both claim success.

2. Define the document state machine

Track each immutable document version independently, and maintain a separate canonical document record pointing at the currently eligible version.

• DISCOVERED: source event received; canonical identity, tenant and source revision recorded.
• FETCHING: source bytes and authenticated metadata are being captured.
• STAGED: immutable source snapshot, hash, lineage and metadata persisted.
• PROCESSING: extraction, parsing, chunking and embeddings in progress.
• INDEXING: version-specific chunks are being upserted to a non-serving candidate.
• VALIDATING: verify chunk manifest, search visibility, citation targets, ACLs, completeness, source revision and security.
• READY: all validation checks pass, but the version is not yet user-visible.
• PUBLISHED: a guarded, atomic serving-pointer promotion makes the complete eligible version visible.
• SUPERSEDED: a newer authorized version has been promoted; the old version is no longer eligible for ordinary current queries but may be retained for permitted historical queries.
• DELETION_REQUESTED: tombstone/deny-serving flag committed immediately, overriding any READY/PUBLISHED version. No new query may retrieve its content.
• DELETING: vector, source-object, derivative and cache purges are underway.
• DELETED: deletion has been verified across the defined serving/storage scope; record audit evidence and known retention exceptions. Do not equate a successful delete API request with confirmed deletion.
• FAILED / QUARANTINED: a recoverable failure or integrity/security exception; preserve the last safe published version only if not revoked. Retry via explicit authorized recovery transitions.

Allowed transitions, not free-form status updates: DISCOVERED → FETCHING → STAGED → PROCESSING → INDEXING → VALIDATING → READY → PUBLISHED → SUPERSEDED. From any non-deleted state, a valid deletion command can move the canonical document to DELETION_REQUESTED; then DELETING → DELETED. Failures branch to FAILED or QUARANTINED with controlled retry. A superseded version must never automatically become PUBLISHED again; an authorized rollback must create an auditable promotion decision and pass current permission/effective-date checks.

3. Persist state and evidence separately

• Document aggregate: tenant_id, canonical_document_id, active_version_id, lifecycle_generation, serving_blocked, deleted_at, updated_at.
• Immutable version record: document_version_id, source_system, source_record_id, source_revision, content_sha256, manifest_sha256, acl_policy_version, status, expected_chunk_count, visible_chunk_count, supersedes_version_id.
• Append-only transition ledger: event_id, aggregate_id, version_id, generation, from_state, to_state, event_type, actor/service, occurred_at, recorded_at, workflow_run_id, correlation_id, idempotency_key, reason, evidence_uri, and a schema version.
• Transition ledger records support investigation, but true compliance-grade immutability may need restricted write privileges, tamper-evident hashes/signatures, access logs, immutable retention and independent backups; a writable application table alone is not tamper-proof.
• Use one database transaction to compare the expected state/generation, update state and append its audit event. Add a transactional outbox event in the same transaction; a separate relay dispatches work after commit.
• Make workers at-least-once safe: deduplicate by event/idempotency key, use versioned chunk IDs, and fence retries with generation/sequence checks. Out-of-order older events must not overwrite a newer state.
• External vector databases and object stores are not in the SQL transaction. Use reconciliation, receipt checks, retry-safe compensating actions, and explicit observable DELETING / VALIDATING states instead of claiming cross-system atomicity.

4. Publication, supersession and deletion invariants

• Publish: only READY with verified full chunk manifest, current source authorization, ACLs and document generation can promote the canonical active-version pointer. Retrieval must apply the active-version filter and current tenant/ACL rules on every access path.
• Supersede: promote the validated new version and demote the old version in a single guarded metadata operation; preserve citation/source lineage and historical eligibility only where policy permits.
• Delete: commit the deny-serving tombstone before asynchronous erasure. Every retrieval and cache path must respect it, including delayed requests and any replica. Reject later indexing and publish events for the revoked generation.
• Deletion completion: check vector IDs, text chunks, stored source snapshots, previews, reranker/answer caches and scheduled backfills; document any approved immutable backup or legal-hold exception honestly.
• Recovery: a failed attempt never skips required validation. Replaying history should reconstruct both the last committed state and pending external actions; manual overrides require named approvers and reasons.

5. Beginner-friendly numerical incident

• Document DOC-42, version v1, has 80 verified chunks and is PUBLISHED. An update creates v2 with 100 expected chunks.
• After a worker crash, 70 of 100 v2 chunks are indexed: completeness = 70/100 = 70%. Since publication requires 100% of manifest IDs plus integrity/ACL checks, v2 stays INDEXING and is invisible; v1's 80 chunks remain eligible.
• A retry writes the 30 missing chunks idempotently. Validation confirms 100/100; a guarded transaction records v2 READY → PUBLISHED, changes active_version = v1 → v2, and records v1 → SUPERSEDED.
• At 14:00, source owner deletes DOC-42. The system commits a tombstone by 14:00:02 (2 seconds later); all query paths now reject DOC-42 even though physical vector deletion takes 4 minutes. A stale worker's publish(v1) event is rejected because its generation is old.
• By 14:04, deletion reconciliation confirms 180/180 stored vector IDs removed (80 from v1 + 100 from v2), plus verified associated caches/objects within the specified scope. Only then transition to DELETED. Historical events remain in a protected audit ledger with minimal content identifiers according to retention policy.
• In a batch of 1,000 lifecycle transitions, assume 997 carry valid audit events and evidence links; audit completeness = 99.7%. A target of 100% required events means three missing entries are incidents to investigate, not acceptable silent gaps. This is an illustrative KPI.

6. Trade-offs and edge cases

• DB state + append-only events is straightforward for audits, but increases schema and operational complexity. Event-sourced workflow engines offer durable histories/replay; still implement authorization and external-service reconciliation.
• Strict gates improve safety but delay publication and raise temporary unavailability. Avoid bypassing validation for urgent changes; offer controlled emergency approval with full audit records.
• Exactly-once end-to-end is usually unrealistic across a queue, SQL database and vector store. Prefer at-least-once delivery, idempotent side effects, generation fencing and observable completion.
• Deletion vs evidence retention: audit metadata may remain, while document content and embeddings must follow contractual/legal deletion requirements. Record retention exceptions and separate "blocked from serving" from "physically purged".
• Concurrent updates and deletes: deletion wins immediately; rejected late retries are logged. A legitimate restore is a new authorized ingestion/generation, never a resurrection by retry.
• Clock skew: use database sequence/order for committed transitions; preserve separate source and observed timestamps. Do not determine causal order using wall-clock timestamps alone.
• Workflow history cost: high-volume chunk events can explode history size. Record meaningful document transitions with manifest pointers and aggregate counts; store per-chunk details in separate partitioned evidence storage.

7. Prevent recurrence

• Define allowed transitions and invariants as code with guarded compare-and-swap operations, not free-form worker status updates.
• Test crashes before/after SQL commit, before/after outbox dispatch, halfway through indexing, during version promotion, and during deletion. Inject duplicate, reordered and delayed messages.
• Alert on READY without verified manifest, PUBLISHED with missing source/ACL evidence, stuck DELETING, tombstoned content returned by any retrieval path, stale-generation publishes, or gaps in the transition ledger.
• Run scheduled reconciliation between source inventory, SQL aggregate/version state, actual vector IDs, object-store contents and live serving/citation results. Export a deletion evidence report.
• Review transition overrides, actor permissions, audit-retention policies, outbox backlog and deletion SLAs with security and document owners.

Interview takeaway: Treat production RAG ingestion as a guarded state machine with durable transition evidence, not a collection of loosely coordinated jobs. Publication requires a verified, atomically selected version; deletion must immediately block serving and only complete after independent purge verification.

References

• Temporal: Event History — https://docs.temporal.io/encyclopedia/event-history
• AWS Prescriptive Guidance: Transactional outbox pattern — https://docs.aws.amazon.com/prescriptive-guidance/latest/cloud-design-patterns/transactional-outbox.html
• AWS Step Functions: Viewing execution details — https://docs.aws.amazon.com/step-functions/latest/dg/concepts-view-execution-details.html