How do you safely fail back multi-agent memory-compaction workflows to the primary region after disaster recovery?

Failback means transferring live AI-agent workflows from a recovered disaster-recovery (DR) region to the original primary region without reactivating stale controllers, losing committed memory, or duplicating delayed tool and compaction effects. It is a separate operation from failover: the DR region may already be authoritative, while the old primary has outdated memory, checkpoint and job-ledger records. Simply switching DNS or restarting old Celery workers can create a new split brain.

Where this fits in the curriculum

Question 257 establishes fencing against stale controller generations; 258 reconciles queued and retried work; 259 verifies an initial controller cutover; 260 restores authority and workflow state during an outage. Question 261 is about the reverse migration after service has recovered, including state convergence, the final authoritative switch, paused agent workflows and post-failback verification. General chaos engineering (Question 209) addresses fault injection broadly rather than this operational sequence.

Beginner-friendly numerical production example

A LangGraph + Celery customer-support platform has 10,000 active workflows. At 10:05, region A fails. Region B takes over with failover epoch 44, processes 800 new compaction operations, and advances 300 workflow checkpoints. Region A returns at 12:00, but its database still has epoch 42, none of those 800 completed operations, and 50 previously reserved v7 tasks.

The team cannot treat region A as primary merely because it has recovered. First copy or rebuild the 800 committed effects and 300 checkpoint advances from authoritative region B evidence (some operations may correspond to the same workflows; they are separate counts), reconcile the ledger and outbox, and keep A's old v7 tasks fenced. Suppose the cutover ledger initially finds 790/800 verified compaction effects, 10 ambiguous downstream index receipts, and 300/300 checkpoint advances. Failback is blocked, even though 98.75% of effects are verified. After inspection, all 10 are resolved or explicitly quarantined with affected workflows kept read-only; only then consider authorization for eligible workflows.

The safe switch grants region A an entirely new epoch 45 after isolating B from further writes for the transferred scopes. Even an old region-B job with epoch 44 must fail at the write boundary. The numbers and acceptance criteria are illustrative; a production implementation must determine its own transaction boundaries, risk thresholds and objectives.

A staged failback protocol

1. Confirm the source of truth. Designate region B authoritative until the handover is complete. Inventory canonical facts, memory revisions, LangGraph checkpoints/pending writes, operation ledger, outbox receipts, policy hashes, tenant keys and paused agent threads. Reject a failback plan if the primary's restored data is assumed current without comparison.
2. Build a consistent target. Replicate from B to A using database change streams or snapshots plus logs. Maintain source commit order and identifiable high-watermarks; don't infer consistency from timestamps on different machines. Rebuild derived vector indexes from authoritative memory, checking version/content hashes and tenant partitions.
3. Prove convergence by cohort. Compare ledger dispositions, canonical-memory revisions and hashes, checkpoint lineage, pending interrupts, tool-side-effect receipts and outbox lag for a defined cohort. Sampled data-quality tests complement, but cannot replace, hard invariants. Check both normal traffic and delayed retries.
4. Fence the final handover. For each tenant/workflow scope, temporarily stop or serialize writes in B, establish a durable final watermark, catch up A, and atomically grant a new epoch greater than any previously accepted epoch to A using an independent authority mechanism. If exclusivity or monotonic authority cannot be proved, keep the cohort on B or in read-only mode. A routing flag or DNS switch by itself does not enforce fencing.
5. Route progressively and drain safely. Start with a small tenant/workflow cohort. A Celery retry issued in B retains its logical operation ID; if the operation already committed, return the recorded result. Otherwise replan from the latest checkpoint under epoch 45. Never replay tool calls that have unknown non-idempotent external effects.
6. Verify no dual writers. Inject old-region worker wake-up, network partition and delayed acknowledgement; assert zero accepted epoch-44 or epoch-42 writes after scope transfer. Validate protected facts, complete ledger accounting, p95/p99 task latency, indexing receipts and customer-visible agent outcomes.
7. Keep a viable reverse path. If A regresses, transfer authority back to B using a newer epoch 46, not epoch 44. Reverse replication/convergence must be verified before B resumes writes. Keep immutable snapshots, operator approvals and a runbook until the rollback observation window closes.

Acceptance gates and an illustrative calculation

For the initial 800 compactions, verified closure rate = verified terminal dispositions / 800. With 790 verified and 10 unresolved, closure is 98.75%, not 100%; these 10 must remain explicitly unresolved rather than assumed successful. A per-cohort failback gate could require 800/800 accounted for, zero unauthorized writes, zero verified loss of protected canonical facts, all required checkpoint lineages consistent, and downstream receipts resolved or safely isolated. A 100% closure count does not imply zero errors: quality and authorization checks remain independent.

Measure actual RTO of failback separately from disaster failover RTO. For example, if a cohort enters write-freeze at 12:10 and resumes verified writes at 12:14, that cohort's write-unavailability is 4 minutes; report any degraded-read interval and end-to-end customer latency separately. RPO must cover canonical memory and ledger, not only the database volume.

Production failures, trade-offs, and mistakes

• Stale primary becomes authoritative: A accepts old v7 task attempts. Require newer epoch fencing checked by the actual storage and side-effect sink.
• Two writable regions during handover: partial network isolation defeats a naive distributed lock. Keep a single durable authority per scope, explicit stop/catch-up/transfer, and tested partition behavior.
• Checkpoint ahead of memory: agent resumes with facts absent from A. Verify lineage and committed-memory revision before resuming; quarantine mismatches.
• External API side effects duplicated: task retries resend a payment or notification. Reconcile external receipts and use provider idempotency keys where supported; otherwise escalate ambiguity.
• Search index divergence: regenerated summaries appear while canonical facts lag. Rebuild and version derived indexes; verify tenant boundaries.
• Latency versus consistency: a short write-freeze improves correctness but affects availability. Finer cohort-by-cohort transfer limits blast radius, at the cost of coordination complexity.
• Unnecessary rush: recovery at B may be stable; don't fail back merely because A is reachable. Require capacity, policy, security and operational readiness.

Monitoring, drills and governance

Monitor authority mismatches, accepted stale writes (target zero), replication lag in commits and seconds, ledger ambiguity age, checkpoint-memory revision mismatch, outbox/index receipts, paused workflow resumption, protected-fact-loss incidents, and p95/p99 agent latency per cohort. Test late retries, lost acknowledgements, worker crashes and rollback-to-B with a fresh epoch during scheduled exercises. Record who approved each cohort, source and target watermarks, test results, temporary degraded-service exceptions, and the final sign-off. Verify risk-based retention and tenant-specific controls.

References

• AWS Well-Architected, Testing disaster recovery implementation: https://docs.aws.amazon.com/wellarchitected/latest/framework/rel_planning_for_recovery_dr_tested.html
• AWS, Testing disaster recovery: https://docs.aws.amazon.com/whitepapers/latest/disaster-recovery-workloads-on-aws/testing-disaster-recovery.html
• LangGraph, Persistence: https://docs.langchain.com/oss/python/langgraph/persistence
• Celery, Tasks and acknowledgements: https://docs.celeryq.dev/en/stable/userguide/tasks.html
• PostgreSQL, continuous archiving and point-in-time recovery: https://www.postgresql.org/docs/current/continuous-archiving.html