What is chaos engineering in multi-agent systems?

Chaos engineering is the practice of deliberately introducing controlled disruptions into a running system to learn whether it remains reliable. In a multi-agent system, it tests not only whether agents recover from failures but also whether their coordination, shared state, and final decisions remain correct.

Beginner-friendly example: API test generation

Imagine a production-like workflow with four specialists: a requirements agent, test generator, security reviewer, and supervisor. The team uses LangGraph for orchestration and Celery workers for execution.

A normal test confirms each agent works when its dependencies are available. A chaos experiment asks: What happens if the security-review worker disappears just after the generator saves its test suite?

A safe outcome is that the suite remains saved but unapproved, the interrupted review is retried or routed according to policy, and the system never publishes an unchecked result.

Form hypothesis → Define safety limits → Observe steady state → Inject one controlled disruption → Measure behavior → Stop or roll back → Learn and improve

Step-by-step numerical example

Suppose the team defines the following experiment in a staging environment:

• Run 100 representative test-suite workflows under normal conditions.
• The baseline produces 96 approved and valid suites, 4 legitimate failures, and p95 completion latency of 30 seconds.
• In the next comparable batch, terminate one security-review worker for 60 seconds while other workers remain available.
• The experiment produces 94 approved and valid suites, 6 safely failed or deferred workflows, zero unreviewed suites marked approved, and p95 latency of 43 seconds.

The approval rate changed from 96% to 94%, a 2-percentage-point decrease. The p95 latency increased by 13 seconds. The safety invariant held because no unreviewed suite was approved.

The experiment reveals a latency and availability weakness. The next investigation might assess retry scheduling, worker redundancy, and review-time deadlines. These numbers are illustrative, not expected production results.

What makes this chaos engineering?

A controlled experiment has:

1. A hypothesis: For example, losing one reviewer worker should not lead to unapproved results being published.
2. A steady-state measurement: Valid outcomes, queue age, p95 completion time, and failed workflow count.
3. A bounded fault: A known worker, dependency, network path, or resource limit is intentionally disrupted.
4. A blast-radius limit: Restrict affected traffic, tenants, duration, and environment.
5. A stop condition: Abort if safety, data integrity, or agreed service thresholds are breached.
6. A learning loop: Record actual behavior, diagnose gaps, implement fixes, and rerun.

Start with staging or isolated canary traffic. Production experiments require explicit ownership, observability, safeguards, and a clear rollback path.

Useful experiments for multi-agent systems

• Agent or worker loss: Terminate one worker mid-handoff; check that completed artifacts survive and task ownership is recovered safely.
• Slow model provider: Introduce latency or a bounded rate-limit response; check deadlines, queuing, and safe degradation.
• Message duplication or delay: Verify idempotency and rejection of outdated handoffs.
• Shared-state disruption: Temporarily deny access to the artifact store; ensure partial work is not silently treated as complete.
• Network partition: Isolate a subset of workers; look for split ownership, stale writes, or conflicting outcomes.
• Coordinator failure: Restart the supervisor during a branch join; verify durable progress and correct final status.

Do not introduce faults into real payment, deletion, or notification operations without sandboxing and side-effect controls.

Measuring results

Measure both reliability and semantic correctness:

• Percentage of workflows reaching a valid final state within a deadline.
• Percentage of incomplete or unreviewed artifacts incorrectly approved—ideally zero.
• Recovery duration, stale-write conflicts, duplicate external actions, and queue growth.
• Token/API costs and retries introduced by the disruption.

An agent continuing to generate text does not mean the workflow recovered successfully.

LangGraph and Celery considerations

In LangGraph, use workflow IDs, state checkpoints, and node-level traces to observe interruption and resumption. Checkpoints alone do not guarantee external side effects are exactly once; verify those separately.

With Celery, test worker termination, acknowledgments, broker redelivery, and task idempotency under the actual deployment settings. Avoid interpreting a retried task as proof that the previous attempt had no external effect.

Chaos engineering vs. failure-scenario testing

Failure-scenario testing typically verifies specific expected responses to injected faults, often in deterministic tests. Chaos engineering examines system-wide behavior under realistic disruptions against measurable steady-state hypotheses, including unexpected interactions between agents and dependencies. They are complementary.

In short

Use controlled, observable chaos experiments to verify that agent teams preserve correct decisions, durable state, and safe recovery during disruptions. Start small, protect production data and side effects, and turn findings into repeatable reliability improvements.