How do you manage agent communication during network partitions?

A network partition happens when some agents or services cannot communicate with others, even though their processes may still be running. A multi-agent system must handle this uncertainty without duplicating important actions or accepting conflicting updates.

Beginner-friendly example: API test generation

A coordinator delegates security-test generation to Agent A, but the network connection breaks after A receives the job.

• The coordinator cannot reach A and does not know whether work started.
• Agent A may still be generating test cases.
• Sending the same job to Agent B immediately could create two competing outputs.

The correct response is not simply to retry every request. Preserve the logical task ID, mark the outcome unknown, and follow a recovery policy.

Detect lost connectivity → Preserve task state → Restrict risky writes → Wait or reassign safely → Reconcile after reconnection

Simple numerical example

Imagine three workers share a job system:

• Two workers can reach the authoritative database.
• One worker loses database access but can still call an LLM.
• All three were previously capable of generating test cases.

If the isolated worker continues publishing local results while the other two process reassigned work, the system may end up with duplicate suites. Instead, the isolated worker must not commit authoritative completion without validating its current ownership.

The two connected workers may continue processing tasks through the authoritative store, but only if they satisfy the system's normal lease, fencing, and authorization rules. Two out of three reachable workers is not automatically a safe quorum; a quorum matters only when supported by the actual coordination protocol.

How should communication behave during a partition?

1. Use finite deadlines and health signals: Treat silence as uncertainty, not proof that the remote agent is dead.
2. Persist messages and task progress: Prefer durable queues and workflow checkpoints rather than memory-only state.
3. Assign stable logical task IDs: Recognize retries and redeliveries as attempts at the same work.
4. Protect ownership: Coordinate claims through an authoritative store and reject writes from obsolete owners using fencing or equivalent safeguards.
5. Separate safe from unsafe work: Agents may continue independent computations, but operations requiring global coordination or irreversible side effects should pause if authority cannot be established.
6. Bound retries and queues: Use backoff, deadlines, and backpressure to avoid a reconnect storm.
7. Reconcile after healing: Compare task status, versions, and output artifacts before accepting late responses or retrying missing work.

Consistency vs. availability

During a partition, a distributed system cannot always provide both fully consistent shared state and uninterrupted writes. For critical actions such as approving a security review, choose a conservative policy that requires authoritative state. For temporary draft generation, local work may be acceptable if the result remains explicitly provisional.

This tradeoff depends on the system's architecture, not a universal rule that every agent must stop.

LangGraph and Celery considerations

In LangGraph, checkpoints can preserve workflow progress, but coordination among disconnected workers still needs an authoritative state and recovery design. Store request IDs and task versions so resumed branches can reject stale updates.

With Celery, broker connectivity loss, worker availability, and result-backend connectivity are distinct failures. Delivery and redelivery behavior depend on broker and task acknowledgment settings. Test actual failure scenarios; do not assume that a timeout cancels remote execution.

Common mistakes

• Treating missing heartbeats as proof an agent has stopped.
• Allowing two disconnected coordinators to approve conflicting results.
• Assuming a durable queue guarantees exactly-once execution.
• Retrying so aggressively that the recovered system is immediately overloaded.
• Automatically trusting results generated offline without version or permission checks.

In short

Handle network partitions by preserving identity and state, limiting unsafe actions during uncertainty, and reconciling work after connectivity returns. Reliability means avoiding both lost work and contradictory outcomes—not merely reconnecting as quickly as possible.