How do you conduct postmortems for failures in multi-agent systems?

A postmortem is a structured review after an incident that explains what happened, why the system allowed it, how users were affected, and what changes will prevent recurrence. In multi-agent systems, investigate not just failing workers but also incorrect reasoning, missing handoff information, and unsafe supervisor decisions. The goal is learning and improvement—not blaming a person or a single agent.

Beginner-friendly example: API test generation

A requirements agent extracts API rules, a test generator writes test cases, a security reviewer checks coverage, and a supervisor approves publication. During an incident, some suites were published even though security review had timed out.

The postmortem should explain why the approval gate accepted a missing review result, which workflows were affected, and which safeguards failed.

Collect evidence → Build timeline → Quantify impact → Trace causal chain → Identify contributing conditions → Assign corrective actions → Verify changes

Step-by-step numerical example

Suppose logs and artifact records show:

1. At 10:00, a new supervisor version was deployed.
2. At 10:12, the security-review service began timing out.
3. From 10:12 to 10:32, 120 suites were generated.
4. 18 suites were marked APPROVED without a valid security-review outcome.
5. At 10:32, automatic approval was disabled; the 18 outputs were quarantined and independently rechecked.

The affected-output rate was 18 / 120 × 100 = 15% during the incident window. The observed exposure interval was 20 minutes. These measurements quantify impact, but do not by themselves establish why approval gating failed.

A later replay shows the supervisor treated a missing reviewer field as an implicit success. That is an application logic defect; reviewer timeouts and inadequate integration tests were contributing factors.

What belongs in a useful postmortem?

1. Executive summary: Customer-visible failure, date, current status, and why it mattered.
2. Impact: Number of affected workflows, missed deadlines, invalid or unsafe artifacts, exposure duration, and any required remediation.
3. Timeline: Evidence-backed events from deployment through detection, containment, recovery, and verification.
4. Detection and response: Which alert fired, what responders initially saw, and where observability or runbooks were insufficient.
5. Causal explanation: How an upstream failure propagated through messages, state, validation, and supervisor decisions.
6. Contributing factors: Ambiguous handoff contracts, stale artifacts, retry behavior, implicit defaults, or missing tests.
7. What worked: Safeguards that successfully limited harm, including quarantines and independent checks.
8. Corrective actions: Concrete changes with owners, priority, due dates, and measurable verification criteria.
9. Open questions: Preserve uncertainty when evidence is incomplete; do not invent a single root cause to make the report look tidy.

How do you find the real cause?

Follow the data lineage:

• Check the original requirements and artifact versions.
• Inspect the review request, its timeout/error result, and the supervisor input.
• Reproduce the exact transition that changed an incomplete review into APPROVED.
• Test the proposed fix with the same failing case and related edge cases.

Separate the trigger (reviewer timeout), the defect (missing outcome interpreted as approval), the impact (unreviewed suites published), and the missing safeguard (no final explicit approval-state assertion). A multi-agent failure often has more than one contributing condition.

What corrective actions are effective?

• Require explicit PASS from every mandatory reviewer; missing, timed-out, or invalid outcomes cannot approve publication.
• Add contract tests for absent fields, stale artifact versions, and failed handoffs.
• Add integration tests reproducing the incident, plus a production-safe canary for approval logic.
• Create a metric and alert for published artifacts without valid review evidence.
• Audit and repair all affected outputs, not only future runs.
• Validate that the fix works under retries, redelivery, and provider timeouts.

An action such as “agents should communicate better” is too vague to verify.

LangGraph and Celery considerations

In LangGraph, preserve workflow IDs, node/edge transitions, checkpoint versions, and review-state decisions. Replay carefully: recovering a checkpoint does not undo external publication side effects.

In Celery, correlate task attempts, queue delays, timeout records, acknowledgments, and redeliveries with the same workflow. A worker returning successfully does not establish that the produced artifact passed quality review. Avoid exposing tenant data or secrets in incident documents.

Postmortem vs. incident response

Incident response focuses on immediate detection, containment, recovery, and communication. The postmortem is the subsequent evidence-based learning process, producing verified engineering actions and follow-up checks.

In short

Write a blameless, evidence-based account of the failure and its contributing conditions. Convert findings into owned, testable safeguards, and confirm they prevent the original incident without creating new problems.