How do you design fault isolation between agents?

Fault isolation in a multi-agent system means designing boundaries so a failure in one agent, tool, or dependency does not automatically disrupt unrelated work. A stalled security reviewer should not exhaust every worker slot or corrupt the state of independent API test-generation tasks.

Beginner-friendly example

An API testing platform runs three specialists:

• Requirements agent: Extracts API rules.
• Functional-test agent: Generates normal test cases.
• Security-review agent: Reviews authorization coverage.

Suppose the security reviewer calls a slow external model provider. Without isolation, its stalled requests can consume all shared worker slots and delay the other two agents. With separate resource budgets and task queues, functional-test generation can continue while the security review is marked blocked or pending.

Separate boundaries → Limit resource use → Contain failures → Preserve independent work → Recover deliberately

Numerical example: the bulkhead pattern

Imagine a worker service can run 12 tasks concurrently.

• Without isolation: All 12 slots are shared. If the security agent occupies all 12 with hanging requests, no functional tests can run.
• With bulkheads: Reserve at most 4 slots for security review and 8 slots for other eligible tasks. Even if all 4 security slots stall, 8 slots remain available for other work.

The split is illustrative. Reserved capacity can reduce utilization during quiet periods, so tune limits using real demand, queue wait times, and service objectives. The effective boundary must include model/API quotas and database connections, not just local threads.

How do you isolate faults?

1. Separate execution resources: Use dedicated worker pools, queues, processes, or containers for failure-prone tasks.
2. Set per-agent concurrency and budgets: Limit simultaneous requests, memory use, token spend, and external API calls.
3. Apply timeouts and circuit breakers: Stop long waits and repeated calls to unhealthy dependencies.
4. Isolate state and side effects: Give each task scoped state, stable IDs, and controlled access to shared artifacts.
5. Define failure propagation rules: Distinguish mandatory dependent steps from independent branches that may continue.
6. Protect shared dependencies: Use connection pools, quotas, and rate limits so one tenant or agent cannot monopolize capacity.
7. Recover safely: Retry or move incomplete work only after checking its current execution state and permissions.

Fault isolation vs. fallback behavior

Fault isolation prevents a broken component from taking down unrelated components. Graceful degradation defines what usable but limited result the user receives during an outage. Circuit breakers prevent repeated requests to a failing dependency. These patterns complement one another; they are not interchangeable.

For example, isolation keeps the functional-test generator available; graceful degradation labels its output security review incomplete; and a circuit breaker stops repeated failing security-review calls.

LangGraph and Celery example

In LangGraph, independent graph branches can carry distinct task state and error handling. A failed review branch should not erase completed generation artifacts. A supervisor can aggregate results and apply a policy for required versus optional steps.

In Celery, use separate queues and appropriately sized worker pools for workloads with different risk or resource profiles. Configure concurrency and limits based on measured task behavior. Routing to different queue names alone does not isolate work if all queues still share one exhausted worker pool or external API quota.

Common mistakes

• Using a single unbounded worker pool for every specialist.
• Treating an agent's logical graph node as a dedicated physical process.
• Sharing mutable state without atomic updates or tenant authorization.
• Failing to limit an external provider shared by all agents.
• Reporting a complete result when a mandatory isolated branch failed.

In short

Isolate execution capacity, state, and dependencies so one failing agent cannot exhaust or corrupt the whole workflow. Combine bulkheads with monitoring, timeouts, circuit breakers, and explicit partial-result policies.