How do you recover agent workflows after a network partition?

Recovering a multi-agent workflow after a network partition means reconciling what each agent actually completed while connectivity was lost, then resuming only work that is still valid and unfinished. Reconnection alone is not recovery: agents may hold conflicting views of task ownership, versions, and results.

Beginner-friendly example: API test generation

A coordinator assigns the security-test agent a job for POST /payments. The connection breaks after the agent starts. During the partition:

• The coordinator marks the result unknown and prepares a replacement attempt.
• The original security agent finishes and saves a local draft.
• Another authorized agent generates a second draft after obtaining a new task claim.

When communication returns, both drafts exist, but only one may be eligible to become the official output.

Reconnect → Read authoritative task state → Collect late outcomes → Compare ownership and versions → Accept or reject results → Resume missing work

Numerical example: deciding which result wins

Suppose logical task payment-security-42 has these attempts:

• Attempt A: Ownership token 11; completed locally during the partition.
• Attempt B: Ownership token 12; obtained a valid new claim and saved its result.

If the authoritative result store enforces fencing tokens, token 12 supersedes token 11. The workflow must reject A's late write even if its generated content looks useful. If no effective fencing was enforced, first investigate whether any conflicting side effects were already committed; do not assume a higher number alone repairs them.

The accepted result must still pass schema, authorization, and quality checks.

A safe reconciliation procedure

1. Re-establish trusted connectivity: Authenticate agents and verify access to the authoritative database, broker, and artifact store.
2. Freeze conflicting writes when needed: Avoid automatically replaying all offline actions while ownership is uncertain.
3. Read durable state: Compare workflow checkpoints, task statuses, current owners, versions, and recorded outputs.
4. Collect and classify late messages: Identify completed, failed, pending, duplicated, and stale attempts using stable task IDs.
5. Resolve conflicts: Apply documented version or fencing rules; reconcile external side effects separately.
6. Resume only missing operations: Requeue unfinished tasks with bounded retries and current permissions; do not blindly rerun completed work.
7. Validate dependencies: Ensure resumed stages consume the accepted version of upstream artifacts.
8. Restore traffic gradually: Drain accumulated queues without creating a retry storm; record the recovery decision for audit.

What if two agents changed the same state?

For shared authoritative state, prefer transactional version checks or a single accepted owner. For independent, mergeable outputs—for example, test cases covering distinct endpoints—the system may combine validated artifacts if their versions and permissions are compatible.

Not every conflict should be merged. Two competing security approvals, account changes, or external writes may require manual investigation.

LangGraph and Celery example

LangGraph: Persist the workflow's logical task IDs and accepted artifact references in checkpoints. After reconnection, compare checkpoint state with the authoritative task/result store, then route only unresolved branches through a recovery node.

Celery: Broker redelivery and worker acknowledgments may have occurred during the outage. Query durable task status and artifacts before requeueing; worker or result-backend reconnection does not prove that a previously timed-out task never ran. Shared claim records and fencing remain important.

Difference from earlier concepts

• Resuming a failed workflow is the general process of continuing from saved progress.
• Managing a network partition focuses on safe behavior while components cannot communicate.
• Post-partition recovery focuses on reconciling divergent attempts, late responses, conflicting versions, and authoritative results after connectivity returns.

In short

Reconcile first, resume second. Trust durable task identity and ownership records, reject stale outcomes, verify external effects, and restart only the work that is truly incomplete.