How do you safely restore AI-agent memory-compaction controller authority and workflow state after a disaster-recovery failover?

**Disaster recovery after a memory-compaction controller replacement must restore a *consistent set of workflow state, committed memory, job dispositions, and writer authority*—not simply restart containers or restore an old database snapshot.** After a backup or regional failover, the restored policy registry can be older than the live controller generation. If stale workers regain write permission, previously retired logic may corrupt valid agent context.

Prerequisites and distinction from earlier lessons

Question 256 selects policy retirement or replacement; 257 prevents split-brain via fencing; 258 reconciles outstanding logical work; 259 validates post-cutover completion and old-worker decommissioning. Question 260 designs and tests recovery after an unrelated infrastructure disaster, especially when restored control-plane records, LangGraph checkpoints, Celery deliveries, memory revisions and external indexes come from different recovery points. This is not the schema-migration rollback covered in Questions 220–224.

A beginner-friendly numerical incident

An AI support service has 10,000 workflows, and policy v8 replaced v7 yesterday. The latest durable, independently protected recovery watermark is 10:00, with fencing generation 42; at 10:04 the surviving service has advanced to generation 43 and recorded 120 new memory revisions. At 10:05, a region fails. The database backup restores a 10:00 state, while a separate broker retains 150 task deliveries created between 10:00 and 10:05. A naive restore reactivates generation 42; 40 delayed v7 deliveries might appear authorized if their tokens match restored records.

Safe response: keep all mutation workers fenced and freeze broker dispatch; establish an authority epoch 44 (greater than any generation ever issued in surviving systems), recover the 10:00 consistent state, discover the 120 post-watermark revisions through independently durable logs/outbox or cross-region replication, then reconcile 150 deliveries by logical operation ID and current checkpoint. Suppose 100 deliveries correspond to already recovered committed operations, 30 remain needed, and 20 have indeterminate external effects: do not execute the first 100 again, replan the 30 under epoch 44, and quarantine the 20. The numbers are illustrative, and recovery of the 120 changes is possible only if those records were independently preserved. Otherwise declare data loss against the agreed recovery-point objective (RPO), notify owners and reconstruct what can be proven from authoritative inputs.

Recovery objectives and scope

• Set RPO, the maximum tolerable data loss (e.g., 5 minutes), separately for canonical facts, checkpoint state, operation ledger and external index. A five-minute database RPO is insufficient if the operation ledger can lag thirty minutes.
• Set RTO, the target time to resume a *safe level of service* (e.g., 30 minutes). Do not count unsafe writes or hidden missing context as successful recovery.
• Treat policy/fencing authority as security-critical. Never decrease a fencing token on restore. Maintain a durable, monotonic source of writer generations outside the failed recovery domain or use a separate higher-level failover epoch with an authoritative single-writer election; if you cannot prove exclusivity, operate read-only until resolved.
• Preserve tenant isolation, access controls, encryption keys and audit logs; restored data must not bypass tenant-scoped authorization.

Step-by-step restore runbook

1. Contain the incident: disable automatic compaction and new writes; isolate the failed region and revoke writer credentials where possible. Avoid two concurrently writable replicas.
2. Establish authority: elect a single recovery controller using a quorum/coordination mechanism appropriate to the infrastructure. Obtain a new epoch/fence above all known issued values; this requires an independently durable upper bound. If no trustworthy bound exists, use a new unique epoch namespace and enforce it at every write sink, rather than guessing a larger integer.
3. Restore coordinated state: recover canonical source memory, LangGraph checkpoints and pending writes, policy-version registry, job ledger and outbox to mutually compatible recovery points. Record snapshot IDs, log sequence numbers, timestamps and hashes; do not assume cross-system wall-clock timestamps form a consistent snapshot.
4. Rebuild derived state: treat vector indexes, caches and compaction projections as rebuildable where the canonical information survives. Reconcile receipt IDs and content hashes; never promote an index as canonical just because it is newer.
5. Reconcile old and new deliveries: correlate every logical operation, not just task ID, with committed memory revisions and the latest checkpoint. Classify as DONE, REPLAN, SUPERSEDED, CANCELLED or QUARANTINED. Replay only idempotent actions after checking current state.
6. Validate invariants before unfreezing: a stale-epoch write must fail at the storage boundary; protected canonical facts must remain intact; each logical operation must have one effective mutation at most. Explicitly test an old-region worker waking up after failover.
7. Resume gradually: open a small cohort under the new authority, monitor lag, quality and SLO burn; expand by tenant/workload after evidence-based checks. Keep human approval for irreversible compactions until confidence improves.
8. Reconcile after recovery: audit delayed deliveries, late external receipts, unresolved contexts and client-visible outcomes. Publish actual RPO/RTO and a post-incident corrective-action list.

Failure modes and trade-offs

• Backup restores a lower fence: older workers may become authorized. Separate durable failover epochs and enforce fencing at every write.
• Broker replay without ledger: repeated delivery may trigger double compaction. Use transactionally recorded logical IDs and an outbox for side effects.
• Checkpoint newer than canonical memory: agents can reference facts that are absent or already removed. Validate lineage and revision compatibility; quarantine mismatches.
• External vector index newer than relational memory: search retrieves phantom or cross-version content. Rebuild or reconcile the index from authoritative data.
• Region isolation is incomplete: both sites accept writes. Require demonstrable single-writer fencing, not just load-balancer failover.
• Faster RTO versus stronger consistency: read-only fallback may exceed the target RTO, but protects canonical facts; classify degraded recovery honestly.
• Recovery drill unrealistic: testing only worker restarts misses delayed old-generation commits, lost acknowledgements and cross-region replication lag.

How to test and monitor

Quarterly or risk-based disaster-recovery game days should restore an isolated environment and inject: old worker wake-up, partial ledger backup, 5-minute replication lag, lost outbox receipt, duplicated broker delivery, delayed human-paused LangGraph thread, and missing vector index. Measure actual RPO/RTO, stale-write acceptance (target zero), reconciliation closure, checkpoint-memory mismatch rate, protected-fact loss (target zero) and unresolved operation age. A zero-error rehearsal does not prove all possible outages are safe; continue monitoring and audit.

Practical takeaway

Restore authority first, recover canonical state second, reconcile uncertain effects third, and resume compaction last. The order prevents a technically successful recovery from reactivating an obsolete remediation policy.

Sources

• LangGraph persistence and fault tolerance: https://docs.langchain.com/oss/python/langgraph/persistence
• LangGraph durable execution: https://docs.langchain.com/oss/python/langgraph/durable-execution
• Celery task acknowledgements and redelivery: https://docs.celeryq.dev/en/stable/userguide/tasks.html
• PostgreSQL continuous archiving and point-in-time recovery: https://www.postgresql.org/docs/current/continuous-archiving.html
• AWS reliability pillar, disaster recovery: https://docs.aws.amazon.com/wellarchitected/latest/reliability-pillar/plan-for-disaster-recovery-dr.html