How do you manage error budgets and burn-rate alerts in multi-agent systems?

An error budget is the amount of unreliability a service can tolerate while still meeting its service-level objective (SLO). A burn-rate alert warns when that allowance is being consumed too quickly. In a multi-agent system, calculate both from complete, validated workflow outcomes—not individual agent calls.

Beginner-friendly example: API test-generation workflow

A system has requirements, test-generation, and security-review agents. Its SLO states:

99% of accepted API test-generation workflows must deliver a valid, security-reviewed suite within 10 minutes over a rolling 30-day window.

If an agent fails but the workflow recovers and delivers a validated suite before the deadline, the request meets this particular SLO. If the workflow finishes late, omits required checks, or never completes, it fails the SLO.

Observe accepted request → Track validated outcome and deadline → Calculate failed-request ratio → Compare with error-budget allowance → Evaluate short and long alert windows → Investigate and respond

Step-by-step numerical example

Assume 10,000 eligible requests in the 30-day window and a 99% success SLO:

1. Allowed failure fraction: 100% − 99% = 1%.
2. Error budget: 10,000 × 1% = 100 failed or late workflows.
3. Suppose 35 requests have failed the SLO so far. The system has consumed 35% of the budget and has 65 failures of allowance remaining, for this fixed request total.
4. Now suppose a recent 1-hour window has 1,000 eligible requests, of which 40 fail. Its observed error ratio is 40/1,000 = 4%.
5. Burn rate = observed error ratio / allowed error ratio = 4% / 1% = 4×.

At a sustained 4× burn rate, the service would exhaust a 30-day budget roughly four times faster than its normal allocation. This is a rate comparison, not a forecast of actual incidents. For variable traffic, compute budget consumption using the appropriate eligible-request counts and SLO window.

How do burn-rate alerts work?

A useful alert compares both short and long observation windows so teams detect serious incidents without paging on every short-lived fluctuation.

For example, a policy could page when:

• The error-budget burn rate is above 4× over 5 minutes *and* above 4× over 1 hour, with sufficient request volume.
• A slower-burn condition persists across 1 hour and 6 hours, triggering a lower-urgency investigation.

These thresholds are illustrative. Choose windows, burn multiples, sample-volume safeguards, and severities to suit the service's SLO and response capacity. A very short window with only a few requests can create misleading rates.

What must you count correctly?

1. One logical workflow, one outcome: Retries and agent handoffs do not add new denominator entries.
2. Measure user-visible success: A workflow is successful only when all mandatory validations complete within the agreed deadline.
3. Count late results: A correct suite delivered after the deadline violates this latency-based SLO.
4. Include failures consistently: Apply a documented exclusion and cancellation policy; never omit errors solely to improve the metric.
5. Handle pending requests: Do not prematurely classify in-flight workflows; evaluate when completed or when the deadline is missed.
6. Separate technical causes: Tag failure sources such as worker unavailability, model provider errors, schema violations, and unsafe handoffs.
7. Protect result integrity: Deduplicate repeated result events and handle missing or delayed telemetry explicitly.

What should the team do when the budget burns quickly?

• Identify which stage contributes most to invalid or late workflows.
• Check agent and provider health, Celery queue age, retries, and validation failures.
• Reduce risky changes or pause rollouts according to the team's release policy.
• Apply safe mitigations such as provider failover, capacity adjustments, or temporarily disabling optional work.
• Preserve mandatory review and approval stages; never improve SLO metrics by labeling unsafe output as successful.
• Record the incident, learn from the cause, and verify the mitigation against the same SLI.

LangGraph and Celery implementation

In LangGraph, attach a stable workflow ID to nodes, branches, checkpoints, and terminal states. Emit a single authoritative validated-completion event and record the original acceptance time for deadline measurement.

With Celery, correlate task attempts and queue delays to that workflow ID. Worker-level success metrics are useful diagnostics but should not be counted as completed user workflows. A background evaluator can mark requests that exceeded their deadline even when no final event arrives.

Difference from SLA/SLO/SLI definitions

The earlier lesson establishes what service reliability targets mean. This lesson explains how to operate them: calculate remaining error allowance, detect elevated consumption with burn-rate windows, and choose safe operational responses.

In short

Manage error budgets with an explicitly defined workflow-level SLI, consistent counting, and multi-window burn-rate alerts. React to sustained budget consumption while protecting output correctness and required safety checks.