AI Agents
An AI agent is a system that uses a model to choose actions in an environment. Agents often combine reasoning, memory, tools, retrieval, and feedback so they can complete multi-step tasks.
How do you manage error budgets and burn-rate alerts in multi-agent systems?
An error budget is the amount of unreliability a service can tolerate while still meeting its service level objective (SLO). A burn rate alert warns when that allowance is being consumed too quickly. In a multi agent system, calculate both from complete, validated workflow outcomes—not individual agent calls. Beginner friendly example: API test generation workflow A system has requirements, test generation, and security review agents. Its SLO states: 99% of accepted API test generation workflows must deliver a valid, security reviewed suite within 10 minutes over a rolling 30 day window. If an agent fails but the workflow recovers and delivers a validated suite before the deadline, the request meets this particular SLO. If the workflow finishes late, omits required checks, or never completes, it fails the SLO. Observe accepted request → Track validated outcome and deadline → Calculate failed request ratio → Compare with error budget allowance → Evaluate short and long alert windows → Investigate and respond Step by step numerical example Assume 10,000 eligible requests in the 30 day window and a 99% success SLO : 1. Allowed failure fraction: 100% − 99% = 1% . 2. Error budget: 10,000 × 1% = 100 failed or late workflows . 3. Suppose 35 requests have failed the SLO so far. The system has consumed 35% of the budget and has 65 failures of allowance remaining , for this fixed request total. 4. Now suppose a recent 1 hour window has 1,000 eligible requests , of which 40 fail . Its observed error ratio is 40/1,000 = 4% . 5. Burn rate = observed error ratio / allowed error ratio = 4% / 1% = 4× . At a sustained 4× burn rate , the service would exhaust a 30 day budget roughly four times faster than its normal allocation. This is a rate comparison, not a forecast of actual incidents. For variable traffic, compute budget consumption using the appropriate eligible request counts and SLO window. How do burn rate alerts work? A useful alert compares both short and long observation windows so teams detect serious incidents without paging on every short lived fluctuation. For example, a policy could page when: The error budget burn rate is above 4× over 5 minutes and above 4× over 1 hour , with sufficient request volume. A slower burn condition persists across 1 hour and 6 hours , triggering a lower urgency investigation. These thresholds are illustrative. Choose windows, burn multiples, sample volume safeguards, and severities to suit the service's SLO and response capacity. A very short window with only a few requests can create misleading rates. What must you count correctly? 1. One logical workflow, one outcome: Retries and agent handoffs do not add new denominator entries. 2. Measure user visible success: A workflow is successful only when all mandatory validations complete within the agreed deadline. 3. Count late results: A correct suite delivered after the deadline violates this latency based SLO. 4. Include failures consistently: Apply a documented exclusion and cancellation policy; never omit errors solely to improve the metric. 5. Handle pending requests: Do not prematurely classify in flight workflows; evaluate when completed or when the deadline is missed. 6. Separate technical causes: Tag failure sources such as worker unavailability, model provider errors, schema violations, and unsafe handoffs. 7. Protect result integrity: Deduplicate repeated result events and handle missing or delayed telemetry explicitly. What should the team do when the budget burns quickly? Identify which stage contributes most to invalid or late workflows. Check agent and provider health, Celery queue age, retries, and validation failures. Reduce risky changes or pause rollouts according to the team's release policy. Apply safe mitigations such as provider failover, capacity adjustments, or temporarily disabling optional work. Preserve mandatory review and approval stages; never improve SLO metrics by labeling unsafe output as successful. Record the incident, learn from the cause, and verify the mitigation against the same SLI. LangGraph and Celery implementation In LangGraph , attach a stable workflow ID to nodes, branches, checkpoints, and terminal states. Emit a single authoritative validated completion event and record the original acceptance time for deadline measurement. With Celery , correlate task attempts and queue delays to that workflow ID. Worker level success metrics are useful diagnostics but should not be counted as completed user workflows. A background evaluator can mark requests that exceeded their deadline even when no final event arrives. Difference from SLA/SLO/SLI definitions The earlier lesson establishes what service reliability targets mean . This lesson explains how to operate them : calculate remaining error allowance, detect elevated consumption with burn rate windows, and choose safe operational responses. In short Manage error budgets with an explicitly defined workflow level SLI, consistent counting, and multi window burn rate alerts. React to sustained budget consumption while protecting output correctness and required safety checks.
How do you design production dashboards for monitoring multi-agent systems?
A production dashboard for a multi agent system is an operational view that shows whether complete agent workflows are producing valid results, meeting deadlines, and using resources responsibly. Its purpose is to help an operator discover problems and investigate causes—not to display as many charts as possible. Beginner friendly example: API test generation Suppose a requirements agent, test generator, security reviewer, and supervisor work together to create API test suites. The dashboard should answer: Are customers receiving validated suites , not merely text from individual agents? Are any workflows stuck waiting for a handoff, worker, or approval? Are errors coming from the model provider, a specific agent, a tool, or the queue? Are token usage and infrastructure costs rising faster than accepted outputs? Workflow events → Metrics and traces → Aggregation by service and stage → Dashboard → Alert and investigation Step by step numerical example Over the last 15 minutes , a system records: 1. 200 eligible workflows accepted. 2. 180 deliver valid, reviewed test suites within a 5 minute deadline. 3. 12 finish late or invalid, and 8 remain past the deadline. 4. The on time valid completion rate = 180 / 200 × 100 = 90% . 5. Workers execute 300 task attempts including retries, but this is not 300 completed workflows. Suppose the dashboard also shows a p95 end to end latency of 7 minutes , and the review queue has 25 pending tasks . That combination points toward a possible review stage bottleneck. Confirm using queue age, processing spans, worker availability, and traces before blaming the reviewer model. For a time window denominator, define how pending requests are treated: requests still within their 5 minute deadline should not be prematurely counted as failures. Here all 200 requests have either reached a terminal outcome or passed their deadline. What should the dashboard contain? 1. User outcomes: Workflow requests, valid accepted outputs, failed/late/deferred outcomes, and incomplete or unsafe outputs prevented from publication. 2. SLO health: Success rate against the configured objective, end to end latency distribution, and error budget burn. 3. Stage breakdown: Per agent error rates, validation failures, duration, retry count, and handoff related rework. 4. Queue and worker health: Celery queue age, pending task count, worker availability, active slots, and retry spikes. 5. Provider/tool dependencies: LLM/API failures, throttling, request duration, tool timeouts, and database errors. 6. Usage and cost: Input/output tokens, model and tool charges, cost per accepted suite, and remaining spend limits. 7. Coordination: Handoff duration, branch join waits, repeated decisions, and stalled workflow counts. Keep each chart tied to a diagnostic question. Overall metrics should lead to filters and trace level details. A practical dashboard layout Top summary: Valid completion rate, p95 latency, active incidents, and cost per accepted workflow. Reliability section: SLO trend, error budget burn, and failures categorized by cause. Workflow map: Stage latency, retries, and time spent waiting at edges or joins. Capacity section: Queue depth and age, worker utilization, and provider rate limits. Investigation panel: Search by workflow ID to inspect related traces and sanitized artifact versions. Use consistent time windows and clear units. Avoid comparing lifetime totals with five minute rates without labeling the difference. How do you collect data reliably? Assign a stable workflow ID to each user request; attempts and subtasks receive their own IDs linked to it. Emit structured events for acceptance, start/end of stage, validation, retries, handoffs, and final delivery. Instrument metrics, logs, and distributed traces with correlation IDs; adopt OpenTelemetry where suitable. Aggregate high cardinality workflow IDs in trace or log searches, not as unbounded time series metric labels. Deduplicate retried terminal events so one request does not count as several successes. Check missing telemetry and clock skew; an empty dashboard does not necessarily mean the service is healthy. Redact prompts, credentials, tenant data, and sensitive requirements before exporting telemetry. LangGraph and Celery implementation In LangGraph , instrument node spans and graph transitions, recording workflow state, relevant artifact versions, and validation outcomes. Link each specialist's activity to one parent workflow trace. In Celery , collect task runtime, publish to start delay, retry count, worker status, queue age, and result delivery. Join those signals by workflow ID so queue delays are not confused with model inference delays. For tooling, a combination such as OpenTelemetry + Prometheus/Grafana for numeric metrics and a compatible tracing backend is one possible architecture; choose based on operational needs rather than assuming one vendor is mandatory. Dashboard pitfalls Reporting completed agent tasks as successful customer workflows. Showing average latency only, hiding slow tail cases. Displaying provider uptime while omitting invalid final outputs. Paging on every transient retry, creating alert fatigue. Using huge per workflow labels that overload the metrics backend. Treating missing data as evidence that no failures occurred. In short Design dashboards around validated workflow outcomes and fast diagnosis. Start with success, latency, SLO health, and cost; then drill down into agents, handoffs, providers, and Celery workers to locate the real bottleneck.
How do you design an incident response process for multi-agent systems?
An incident response process is a repeatable way to detect, contain, diagnose, resolve, and learn from production failures. For multi agent systems, it must cover both technical outages and semantic failures : agents that are running normally but producing incorrect or unapproved results. Beginner friendly example: API test generation A workflow has a requirements extractor, test generator, security reviewer, and supervisor. An alert shows that several test suites are marked APPROVED even though the security reviewer timed out. This is a safety and integrity incident, not just a slow model call. The immediate priority is to prevent more unreviewed suites from being published , identify affected outputs, and restore a trustworthy approval path. Alert → Declare incident → Assign owner → Contain impact → Diagnose → Recover safely → Verify outcomes → Review and prevent recurrence Numerical example Suppose an incident began at 10:00 : 1. 10:05: An alert detects a spike in invalid approvals. Detection delay = 5 minutes . 2. 10:08: On call confirms the incident and disables automatic approval when review status is missing. Initial containment takes 3 minutes after acknowledgment . 3. 10:20: Engineers deploy a validated fix to the supervisor's approval state checks. 4. 10:30: Affected workflows have been identified, necessary reviews rerun, and the service is restored under monitoring. The illustrative time to restore = 30 minutes from incident start, while detection time = 5 minutes . These are different measures. Teams may also track time to acknowledge, contain, and fully remediate; define each clock consistently. Suppose 80 test suites were delivered during the incident and 12 lacked verified security review. The affected output rate is 12 / 80 × 100 = 15% . All 12 must be quarantined or flagged for revalidation; simply fixing future output is insufficient. A practical incident response process 1. Define severity: Classify incidents by customer impact, incorrect outputs, unsafe actions, data exposure, availability, and deadline breaches. An unreviewed security artifact being approved may be more serious than ordinary latency. 2. Detect and declare: Use workflow level SLO alerts, validation failures, stale state signals, and user reports. Create one incident record with start time and known impact. 3. Assign clear roles: An incident commander coordinates decisions; technical responders investigate; a communications owner updates stakeholders. Small teams may combine roles explicitly. 4. Contain the blast radius: Pause affected publish actions, disable unsafe tool permissions, restrict faulty model routes, or queue work instead of silently approving it. 5. Preserve evidence: Keep correlated trace IDs, event timelines, artifact versions, model/tool versions, and decisions—while protecting prompts, secrets, and tenant data. 6. Investigate contributing causes: Trace the incorrect state and distinguish upstream model/tool failure from bad handoff logic, missing validation, and unsafe supervisor behavior. 7. Mitigate and recover: Roll back the faulty release or deploy a targeted fix. Replay only safe, idempotent steps; revalidate affected outputs before marking them complete. 8. Verify recovery: Check valid workflow outcomes, queue drainage, SLO health, and absence of newly unreviewed approvals under representative load. 9. Communicate: Share known impact, mitigations, next update, and remaining uncertainty without claiming resolution prematurely. 10. Conduct a blameless review: Record the timeline, impact, contributing factors, detection gaps, and concrete preventive actions with owners and deadlines. What should a multi agent runbook contain? For each common failure mode, document: Trigger and symptoms: For example, an elevated rate of security review timeout with approved final status. Immediate safety action: Prevent publication of affected artifacts while preserving evidence. Diagnosis path: Inspect workflow IDs, supervisor transitions, reviewer results, and source versions. Recovery steps: Restore correct approval gating and reconcile all potentially affected artifacts. Escalation: Who can approve rollback, incident severity changes, and customer notifications. Verification checks: A suite cannot be APPROVED unless its required reviewer outcome is present and valid. Runbooks should be exercised before incidents; never assume a workflow rollback reverses external actions such as email delivery or customer visible publication. LangGraph and Celery considerations In LangGraph , correlate node events and checkpoints to a stable workflow ID. A checkpoint may assist resumption, but external side effects require separate reconciliation and idempotency controls. With Celery , investigate broker backlog, task attempts, acknowledgments, redelivery, time limits, and worker health. Prioritize containment of unsafe downstream actions over merely restarting failed workers. How does incident response differ from monitoring and chaos testing? Monitoring dashboards help discover abnormal behavior. Chaos engineering tests resilience through controlled disruptions. Incident response coordinates people and technical controls during a real event, including containment, communication, recovery, and learning. In short An effective incident process protects users first, preserves evidence, fixes the underlying failure, and checks every affected artifact. In multi agent systems, restoration means returning to validated decisions—not just getting agents to run again.
How do you conduct postmortems for failures in multi-agent systems?
A postmortem is a structured review after an incident that explains what happened, why the system allowed it, how users were affected, and what changes will prevent recurrence. In multi agent systems, investigate not just failing workers but also incorrect reasoning, missing handoff information, and unsafe supervisor decisions. The goal is learning and improvement—not blaming a person or a single agent. Beginner friendly example: API test generation A requirements agent extracts API rules, a test generator writes test cases, a security reviewer checks coverage, and a supervisor approves publication. During an incident, some suites were published even though security review had timed out. The postmortem should explain why the approval gate accepted a missing review result , which workflows were affected, and which safeguards failed. Collect evidence → Build timeline → Quantify impact → Trace causal chain → Identify contributing conditions → Assign corrective actions → Verify changes Step by step numerical example Suppose logs and artifact records show: 1. At 10:00 , a new supervisor version was deployed. 2. At 10:12 , the security review service began timing out. 3. From 10:12 to 10:32 , 120 suites were generated. 4. 18 suites were marked APPROVED without a valid security review outcome. 5. At 10:32 , automatic approval was disabled; the 18 outputs were quarantined and independently rechecked. The affected output rate was 18 / 120 × 100 = 15% during the incident window. The observed exposure interval was 20 minutes . These measurements quantify impact, but do not by themselves establish why approval gating failed. A later replay shows the supervisor treated a missing reviewer field as an implicit success. That is an application logic defect; reviewer timeouts and inadequate integration tests were contributing factors. What belongs in a useful postmortem? 1. Executive summary: Customer visible failure, date, current status, and why it mattered. 2. Impact: Number of affected workflows, missed deadlines, invalid or unsafe artifacts, exposure duration, and any required remediation. 3. Timeline: Evidence backed events from deployment through detection, containment, recovery, and verification. 4. Detection and response: Which alert fired, what responders initially saw, and where observability or runbooks were insufficient. 5. Causal explanation: How an upstream failure propagated through messages, state, validation, and supervisor decisions. 6. Contributing factors: Ambiguous handoff contracts, stale artifacts, retry behavior, implicit defaults, or missing tests. 7. What worked: Safeguards that successfully limited harm, including quarantines and independent checks. 8. Corrective actions: Concrete changes with owners, priority, due dates, and measurable verification criteria. 9. Open questions: Preserve uncertainty when evidence is incomplete; do not invent a single root cause to make the report look tidy. How do you find the real cause? Follow the data lineage: Check the original requirements and artifact versions. Inspect the review request, its timeout/error result, and the supervisor input. Reproduce the exact transition that changed an incomplete review into APPROVED. Test the proposed fix with the same failing case and related edge cases. Separate the trigger (reviewer timeout), the defect (missing outcome interpreted as approval), the impact (unreviewed suites published), and the missing safeguard (no final explicit approval state assertion). A multi agent failure often has more than one contributing condition. What corrective actions are effective? Require explicit PASS from every mandatory reviewer; missing, timed out, or invalid outcomes cannot approve publication. Add contract tests for absent fields, stale artifact versions, and failed handoffs. Add integration tests reproducing the incident, plus a production safe canary for approval logic. Create a metric and alert for published artifacts without valid review evidence . Audit and repair all affected outputs, not only future runs. Validate that the fix works under retries, redelivery, and provider timeouts. An action such as “agents should communicate better” is too vague to verify. LangGraph and Celery considerations In LangGraph , preserve workflow IDs, node/edge transitions, checkpoint versions, and review state decisions. Replay carefully: recovering a checkpoint does not undo external publication side effects. In Celery , correlate task attempts, queue delays, timeout records, acknowledgments, and redeliveries with the same workflow. A worker returning successfully does not establish that the produced artifact passed quality review. Avoid exposing tenant data or secrets in incident documents. Postmortem vs. incident response Incident response focuses on immediate detection, containment, recovery, and communication. The postmortem is the subsequent evidence based learning process, producing verified engineering actions and follow up checks. In short Write a blameless, evidence based account of the failure and its contributing conditions. Convert findings into owned, testable safeguards, and confirm they prevent the original incident without creating new problems.
How do you prevent recurring failures in production multi-agent systems?
Preventing recurring failures means converting past production incidents into durable safeguards, tests, and operational improvements so the same failure mode does not keep returning. In multi agent systems, the recurring problem might originate in an agent's output, a handoff contract, shared state, external tools, or the orchestration logic—not necessarily a single model response. Beginner friendly example: API test generation A team runs a requirements extractor, test generator, security reviewer, and publishing supervisor. On three occasions, a timed out security review led the supervisor to approve a test suite anyway. Restarting the reviewer fixes the immediate symptom but not the repeated defect. A sustainable fix requires the publishing supervisor to accept only an explicit, valid PASS result for the correct artifact version, regardless of whether the reviewer is offline or returns incomplete output. Identify repeated failure signature → Find systemic cause → Add a prevention control → Reproduce in automated tests → Roll out safely → Monitor recurrence Step by step numerical example Suppose a service processes 1,000 workflows each week : Week 1: 20 invalid approvals occur. Week 2: 15 invalid approvals occur after workers are restarted. The team then adds an explicit review status gate, version checks, and a regression test. Week 3: 0 invalid approvals occur in 1,000 workflows. Before the fix, the observed recurring incident rates were 20/1,000 = 2% and 15/1,000 = 1.5% . After the fix, the observed rate was 0/1,000 = 0% . This is promising but not proof that recurrence is impossible . Run more tests, monitor longer periods, and consider confidence and workload differences before claiming the risk is eliminated. How do you stop the same failure from returning? 1. Group incidents by failure mode: Compare structured error signatures, invalid output types, artifact versions, and workflow stages—not just alert titles. 2. Separate symptom from cause: Repeated timeouts may be expected; treating timeouts as approval is the preventable defect. 3. Fix at the correct boundary: Validate inputs and outputs at handoffs, require explicit terminal states, and reject stale or incomplete artifacts. 4. Turn incidents into regression tests: Replay representative failing scenarios with deterministic mocks and expected workflow states. 5. Include concurrency and retry cases: Test duplicate deliveries, stale updates, worker restarts, and partial external side effects. 6. Add layered safeguards: Combine schema checks, authorization, approval gates, idempotent operations, and monitoring so one failure cannot silently become a bad final result. 7. Assign ownership: Every corrective action needs an owner, due date, and observable acceptance criterion. 8. Use safe rollouts: Canary or stage changes, watch error signals, and retain a tested rollback procedure. 9. Track repeat failures: Monitor the frequency of the original error signature and unexpected workarounds that hide it. 10. Review effectiveness: Remove ineffective mitigations or strengthen controls if the failure returns. Example prevention contract A publishing step should enforce all of these checks: The test suite has passed required structural and coverage validation. The security reviewer returned PASS , not missing, timed out, or unknown. The reviewer decision references the exact artifact version being published. The publication operation is idempotent for the workflow and artifact version. Any failure produces a visible, non approved state and traceable error reason. This contract makes the approval decision testable independently of the LLM. Measuring prevention effectiveness Track both leading indicators and outcomes : Regression tests passing for known incident patterns. Percentage of handoffs passing contract validation. Invalid approvals and duplicate side effects per 1,000 eligible workflows. Repeat incidents by failure signature and time between recurrences. Corrective action completion and whether each action passed a real verification test. A team can close all its incident tickets while the underlying failure still exists. Measure actual recurrence rather than ticket count. LangGraph and Celery implementation In LangGraph , place explicit validation on state transitions, version aware artifact checks before approval, and bounded failure branches. Checkpoints help restart workflows, but cannot replace safeguards around external side effects. With Celery , make retried tasks replay safe, correlate attempts by stable workflow IDs, and test failures around the critical boundary between database writes and task acknowledgments. Monitor broker redelivery, task retries, and worker pool exhaustion without assuming retries fix semantic errors. Recurring failure prevention vs. a postmortem A postmortem documents what happened, contributing causes, and corrective actions. Recurring failure prevention operationalizes those lessons: changes the system, adds regression coverage, and measures whether the original failure mode stays under control. In short The best prevention strategy converts each significant incident into an enforceable system invariant, automated regression test, and monitored outcome. Do not rely on restarting agents, prompt tweaks alone, or a one time fix without verification.
How do you perform safe rollouts and rollbacks of multi-agent workflows?
A safe rollout introduces a new multi agent workflow version gradually while measuring whether it still produces valid results. A rollback stops sending new work to a faulty version and restores a previously approved version. Unlike a simple code deployment, an agent release may change prompts, models, tools, routing graphs, schemas, or approval rules. Beginner friendly example: API test generation An API test platform uses four agents: requirements extraction, test generation, security review, and approval. Version v2 changes the reviewer prompt and adds a new handoff format. Deploying it to every customer immediately risks propagating a bad review decision. A safer approach tests v2, gives it a small portion of new workflows, monitors independently validated outcomes, and expands traffic only if the release meets predefined criteria. Version artifacts → Test in staging → Route canary traffic → Compare outcomes → Expand gradually or roll back → Audit in flight work Numerical example Suppose v1 has a 98% valid completion rate over a representative baseline, and the team routes 10% of 1,000 new workflows to v2: 1. 100 workflows use v2 and 900 stay on v1. 2. Of the 100 v2 workflows, 94 produce valid, approved suites: 94% . 3. Of the 900 v1 workflows, 882 succeed: 98% . 4. The v2 canary underperforms by 4 percentage points relative to v1. 5. If the release policy requires v2 to be no more than 2 percentage points worse , the canary fails the gate and new traffic returns to v1 . This is an illustrative decision. With only 100 canary runs, sampling uncertainty matters; analyze task difficulty, reliability, and safety failures before drawing broad statistical conclusions. Any critical unsafe approval may justify stopping the release immediately regardless of aggregate rates. What must be versioned together? 1. Graph and orchestration code: Node implementations, conditional edges, retries, and termination rules. 2. Agent instructions: System prompts, tool descriptions, model names, sampling settings, and output contracts. 3. Tool permissions: Allowed operations, credential scopes, API versions, and safety controls. 4. Shared state schemas: Checkpoint format, message types, artifact versions, and migration rules. 5. Evaluation suite: Fixed representative tasks and independent quality/safety checks. 6. Release configuration: Feature flags, traffic percentages, concurrency, and budget limits. Record a release manifest or immutable version ID with every workflow so failures can be attributed to the exact configuration. A practical rollout procedure Preflight: Run contract, integration, security, and regression tests, including past incident cases. Shadow mode when safe: Let v2 process representative inputs without publishing results or invoking unsafe side effects; compare its output with production. Canary: Send a small, scoped share of real eligible workflows to v2 and keep v1 available. Watch gates: Valid completion, security review correctness, p95 latency, queue wait, model cost, retries, and error budget burn. Progressive exposure: Increase to larger shares only after the observation window and quality criteria are satisfied. Stop conditions: Revert on unsafe output, abnormal failure rates, missing telemetry, or sustained SLO deterioration. Keep explicit approval and validation requirements unchanged throughout the deployment. What does rollback actually mean? A rollback is not simply replacing v2 code with v1 code: New workflows: Route them to the known good v1 release. In flight workflows: Decide whether they can finish safely on v2, must be paused, or require a tested migration path. Do not deserialize v2 checkpoints into incompatible v1 graph state. Published or external actions: Rollback does not unsend emails, delete customer visible outputs, or undo payments. Quarantine affected artifacts and perform authorized compensating actions when necessary. Data and schemas: Prefer backward compatible migrations and reversible feature flags; an irreversible database migration may prevent a code only rollback. Model/provider versions: Pin or record versions and test fallback behavior, since remote model changes may not be reversible by redeploying application code. LangGraph and Celery implementation In LangGraph , associate every thread/checkpoint with its workflow release and state schema version. Use release aware routing for new runs, and explicitly test resume behavior before switching a paused run to a different graph version. With Celery , route tasks by compatible release or queue during mixed version operation. Account for tasks already queued or reserved by workers, and make publication operations idempotent. Rolling back a dispatcher does not automatically cancel or downgrade jobs already in the queue. What should you verify after rollback? Confirm that new workflows use v1, affected v2 runs are accounted for, approval gates enforce their rules, and valid completion and latency recover. Compare post rollback traces with the release manifest, and follow up on any customer visible incorrect outputs. Difference from recurring failure prevention Recurring failure prevention strengthens controls so known defects do not return. Safe rollout and rollback limit the impact of new changes and provide a controlled way to retreat when those changes fail. In short Deploy agent workflows as versioned, testable units; gradually expose users; and predefine stop conditions and rollback paths. Always handle checkpoints, queued tasks, shared state, and external side effects explicitly.
How do you manage version compatibility between agents during rolling deployments?
Version compatibility means that agents running different software, prompt, or data contract versions can still exchange information correctly while a deployment is in progress. During a rolling deployment, old and new agents may work simultaneously; without explicit contracts, one agent can silently misread another agent's output. Beginner friendly example: API test generation agents Suppose a requirements extractor produces an artifact that a test generator consumes. Contract v1: The extractor sends {"endpoint":"/payments","auth":"OAuth2"}. Contract v2: It sends {"endpoint":"/payments","auth":{"scheme":"OAuth2","scopes":["pay:write"]}}. An old generator expecting auth as a string might crash, or worse, treat the new object as missing and generate tests without authorization coverage. Version message contract → Check consumer compatibility → Deploy compatible readers → Introduce new writers → Observe mixed traffic → Retire old format Numerical example A service has 10 workers : initially all run v1. The team gradually replaces 2 workers at a time with v2. 1. After the first batch, 8 v1 workers and 2 v2 workers may share queues. 2. If 100 messages are distributed approximately evenly and every worker is eligible, about 20 may reach v2 workers and 80 reach v1 workers; actual broker routing will vary. 3. If v2 immediately emits the new incompatible object shape, any v1 consumer could fail on those messages. 4. Instead, deploy consumers that safely recognize both formats, or route v2 messages only to v2 compatible consumers. 5. After all consumers support v2 and in flight v1 work is drained, new producers can switch to the v2 format. Worker percentages do not by themselves guarantee compatibility. Messages already queued, retries, and resumed workflows can outlive the deployment. How do you design compatible agent contracts? 1. Version every handoff: Include a schema version, stable task ID, source artifact version, and explicit required fields. 2. Prefer additive evolution: Add optional fields first; avoid renaming fields or changing a field's type in place. 3. Use expand–migrate–contract: Deploy readers that accept old and new formats, migrate producers and stored artifacts, then remove legacy support only when safe. 4. Reject unknown breaking versions: Validate messages at the boundary and send incompatible payloads to a controlled failure path instead of guessing. 5. Negotiate capabilities: An agent registry can expose supported contract versions and tool capabilities so the router picks eligible consumers. 6. Version behavior, not only JSON: Model/prompt changes can alter meaning even when the schema remains valid; verify semantic invariants. 7. Keep transformations explicit: If an adapter converts v1 to v2, test the conversion and document any information loss. 8. Protect persisted state: Plan schema migrations for checkpoints, shared memory, and intermediate artifacts with rollback constraints. Example versioned handoff Before processing, a consumer should verify both the supported schema version and that required authorization fields are present and meaningful. What should you test before rolling out? Compatibility matrix: Test v1 producer → v1 reader, v1 → v2, v2 → v1, and v2 → v2. A breaking v2 → v1 path must be blocked or adapted. Contract and semantic checks: Confirm valid parsing and preservation of requirements such as authorization scopes. In flight workflows: Resume old checkpoints with the version they were created for, or use a proven migration path. Delayed and duplicate messages: Verify retries do not unexpectedly cross incompatible worker versions or overwrite newer artifacts. Rollback rehearsal: Ensure the old release can still consume supported data, or pin/route work to the compatible version. LangGraph and Celery considerations In LangGraph , graph state and checkpoint serialization may change alongside node definitions. Record the workflow release and state schema version; do not assume a checkpoint from a new graph can safely resume on an older graph. Check compatibility explicitly. In Celery , rolling worker upgrades leave old and new consumers active. Use versioned task payloads and dedicated queues or routing keys when necessary. A task that runs successfully at the broker level may still produce semantically incorrect output if the consumer misinterprets the payload. Compatibility vs. rollout strategy The previous lesson covers how much traffic to expose and when to roll back . Version compatibility addresses whether old and new agents can safely communicate during that transition . In short Make handoff contracts explicit, introduce backward compatible readers before new writers, test mixed versions, and preserve version aware routing for queued and resumed work. A rollout is safe only when each agent can correctly interpret the data it receives.
How do you migrate in-flight multi-agent workflows to a new version safely?
Migrating an in flight multi agent workflow means moving a job that has already started from one workflow version to another without losing state, repeating unsafe side effects, or changing its intended result. The safest choice is often to let existing jobs finish on their original version; migrate only when there is a tested reason to do so. Beginner friendly example: API test generation An API test generation workflow uses a requirements agent, test generator, security reviewer, and publisher. Version v1 stores an authentication requirement as a string; version v2 stores an object containing the authentication scheme and required scopes. A job currently waiting for security review has a v1 checkpoint . Simply restarting it with v2 code might cause the new reviewer to misread its state—or publish the same test suite twice. Identify running version → Pause at a safe boundary → Snapshot state → Transform and validate → Resume on v2 → Verify outputs and side effects Step by step numerical example Suppose 100 workflows are active during an upgrade: 1. 70 are close to completion and can safely finish on v1 . 2. 20 are paused before security review, at a known migration boundary. 3. 10 are performing external operations and are not safe to migrate yet . 4. The team migrates a canary group of 5 of the 20 eligible workflows . 5. All 5 resume on v2, pass checkpoint validation, and complete without duplicate publication. 6. After checking latency, validation outcomes, and the migration audit records, the team can consider the remaining 15 eligible workflows. A 5/5 success rate is encouraging but too small to prove that all migrations will succeed. The 10 workflows in the side effect stage must be reconciled or allowed to reach a safe boundary first. How do you perform the migration? 1. Inventory active jobs: Record workflow ID, release version, node/stage, checkpoint version, artifact references, and pending external calls. 2. Choose the policy: Finish on v1, migrate at a safe checkpoint, or explicitly cancel and restart a task when that is permitted. 3. Quiesce the workflow: Stop new writes or ownership changes for the specific job before taking the migration snapshot. Use a lock or versioned compare and swap claim so only one runner resumes it. 4. Back up immutable state: Preserve the original checkpoint and release metadata for audit and recovery. 5. Transform state explicitly: Convert v1 fields to the v2 schema using a tested, version specific migration function; reject missing required information instead of inventing it. 6. Validate invariants: Check required fields, handoff recipients, artifact version, tenant authorization, approval status, and workflow stage. 7. Resume with controlled routing: Assign the job to workers that support v2, using a stable workflow ID and new execution attempt ID. 8. Reconcile side effects: Check the durable operation ledger before reissuing publications, notifications, or writes. 9. Monitor and verify: Compare outcomes, retries, semantic quality, and queue delays against the expected behavior. 10. Keep a recovery route: If transformation fails, leave the original checkpoint intact and resume on v1 when safe; never blindly downgrade an already transformed workflow. Tiny state migration example Old state: New state: This transformation is only valid if the scopes really are optional or an empty list is explicitly allowed. If scopes are mandatory and cannot be derived reliably, migration must stop and request an authoritative source instead. The hardest case: external side effects Imagine the publisher saved a suite to a customer facing system but crashed before marking the checkpoint complete. Resuming v2 might replay the publisher and create a second copy. Use a stable publication idempotency key, such as workflow ID + artifact version + operation , and consult the external system or transaction ledger before retrying. Checkpoints record workflow progress, but do not guarantee exactly once external actions. LangGraph and Celery implementation In LangGraph , track graph release and state schema versions alongside thread/checkpoint IDs. Validate that a resumed graph can interpret its state and that the current node still exists with compatible semantics. When this is not true, use an explicit migration adapter or keep the original graph available. With Celery , route migrated tasks to v2 compatible queues and worker processes. Messages already reserved or queued under v1 may still execute, so drain, revoke where appropriate, or reconcile them according to the deployment's acknowledgment behavior. Do not rely on a revoke alone to undo running tasks or external side effects. How is this different from ordinary version compatibility? Version compatibility means old and new agents can exchange messages during a deployment. In flight migration changes the execution version of an already started workflow and must preserve its state, ownership, and side effect history. In short Prefer finishing existing workflows on their original release unless migration is necessary. When moving a running job, pause at a safe checkpoint, validate a versioned state transformation, resume through compatible workers, and verify all externally visible actions.
How do you design backward-compatible changes to multi-agent workflow state?
A backward compatible workflow state change lets an updated multi agent application read existing saved state without losing its meaning or breaking workflows that are still running. This matters when checkpoints, shared state, or agent output structures survive longer than a deployment. Beginner friendly example: API test generation A multi agent workflow stores progress for requirements extraction, test generation, and security review. In version 1, a saved state contains: Version 2 adds an optional review attempts field. A compatible reader treats a missing value in older checkpoints as 0 only when that default is semantically valid . It does not require every existing checkpoint to be rewritten immediately. Define state contract → Make additive change → Test old checkpoints with new reader → Deploy compatible code → Migrate when needed → Retire old state safely Numerical example Suppose 1,000 saved workflow checkpoints exist: 800 use schema v1. 200 use schema v2. The new reader supports both versions. If the new code requires the new field unconditionally, up to 800 / 1,000 = 80% of checkpoints may fail to load. If the field is optional with a justified default and old checkpoints pass validation, those 800 remain usable. However, supporting old data is not sufficient if new code changes what review status means . A previously valid value of PENDING must never be silently interpreted as APPROVED. Practical compatibility rules 1. Version stored state: Include a schema version and the workflow release identifier, keeping them distinct. 2. Prefer additive changes: Introduce optional fields before making them required. 3. Preserve field meaning: Do not silently redefine a status, units, identifier, or authorization flag. 4. Provide explicit defaults: Use defaults only when they preserve intended semantics; missing safety evidence must remain missing. 5. Use version aware readers: Parse, normalize, and validate old formats before running new logic. 6. Avoid destructive renames: Support both the old and new field names during an agreed transition. 7. Plan writer compatibility: Old workers may still write v1 state while new workers write v2; reject stale overwrites using version checks. 8. Test round trips: Confirm loading, updating, and saving state does not discard fields required by other compatible components. Example: unsafe versus safe change Unsafe change: v1 stores review status: "PENDING". v2 replaces it with approved: true as the default whenever approved is absent. This could incorrectly approve old workflows. Safer change: v2 introduces review decision with explicit values such as PASS, FAIL, and NOT REVIEWED. A v1 PENDING state maps to NOT REVIEWED, never PASS. Publication requires a valid PASS tied to the correct artifact version. Testing state compatibility Create a small library of real or sanitized checkpoint fixtures representing: Old and new schema versions. Paused workflows, partially completed branches, and pending approvals. Missing optional fields and unknown extra fields. Replayed, duplicated, and out of order state updates. Corrupt or unsupported states that must fail safely. Validate both structural compatibility and semantic invariants . An object can parse successfully while carrying an incorrect security decision. LangGraph and Celery considerations In LangGraph , changes to state keys, reducers, node identities, and checkpoint content can affect resumed executions. Keep a compatibility adapter where needed, preserve the original checkpoint, and test resumption with representative saved states. New reducer behavior can change how parallel branch updates merge even when field types remain unchanged. With Celery , queued task messages may carry an older state snapshot. Use explicit payload versions and worker routing, and rely on durable version checks when writing results back. A task completing successfully does not prove its stale result is safe to merge. What if compatibility cannot be preserved? Treat a breaking change as a deliberate migration: keep the old reader or worker available, transform state at a verified safe boundary, and stop unsupported resumes with a clear error. Never silently guess missing mandatory information. How is this different from the previous questions? Rolling deployment compatibility concerns communication between old and new agents. In flight migration concerns moving an already started execution to another version. Backward compatible state evolution concerns changing persisted workflow data so old checkpoints and new code can safely coexist. In short Add fields carefully, version persisted state, maintain its semantics, and test old checkpoints against new code. In multi agent workflows, backward compatibility must protect approvals, artifact ownership, and merge behavior—not just JSON parsing.
How do you manage schema migrations for shared memory in multi-agent systems?
A shared memory schema migration changes how multiple agents store and retrieve durable information without losing records, mixing tenant data, or allowing old workers to misinterpret new records. Shared memory may include PostgreSQL rows, documents, vector store metadata, agent generated summaries, and references to prior results. Beginner friendly example An API testing team shares a memory record describing endpoint authentication. The original format contains: The new format needs authentication scopes and supporting evidence: The system must not silently manufacture pay:write if the old record never contained it. Missing authoritative information should remain unknown, or the record should be queued for re extraction. Inventory stored data → Design new contract → Introduce compatible readers → Migrate in batches → Validate semantics → Switch writers → Retire old format Numerical example: gradual migration Suppose there are 10,000 shared memory records : 1. A migration processes 500 records per batch , requiring 20 batches . 2. After 8 batches , 4,000 records have been attempted. 3. If 3,960 pass validation and 40 lack required source evidence, the validated migration rate for attempted records is 3,960 / 4,000 = 99% . 4. Keep the 40 records quarantined or marked MIGRATION REVIEW REQUIRED; do not count them as migrated. 5. Continue migrating unaffected records while a controlled review path resolves the exceptions. This is an illustrative calculation. Real batch sizes should be selected based on transaction size, locking, system load, and recovery requirements. Practical migration steps 1. Inventory every consumer: Identify which agents, prompts, tools, dashboards, retrievers, and background tasks read or write shared memory. 2. Version the record: Track schema version, tenant or access scope, source/provenance, record revision, and migration status. 3. Use expand–migrate–contract: First add compatible fields and dual format readers; then backfill; finally remove the old representation after consumers are upgraded. 4. Keep migrations repeatable: Use idempotent transforms and stable record IDs, so restarting a batch never creates duplicate memories. 5. Prevent stale writes: Use revision comparisons or transactional updates; an old agent must not overwrite a newly migrated record. 6. Preserve meaning: Revalidate facts, timestamps, access permissions, and evidence—not merely JSON syntax. 7. Choose consistency deliberately: Use an authoritative source for memory updates; invalidate or refresh caches and retrieval indexes after committed changes. 8. Plan rollback: Preserve originals or reversible changes while retaining audit logs. Do not assume a code rollback automatically restores modified records. 9. Observe the process: Track attempted, validated, quarantined, failed, and pending records, plus migration latency and error causes. 10. Retire carefully: Remove legacy readers only after confirming that old queued tasks and active agents no longer depend on v1. Vector memory requires extra care Changing only a metadata field may allow a metadata backfill. But changing embedding model or vector dimensions is not merely a schema rename . It may require a new index or collection and re embedding from original source content. During transition, keep retrieval results from different embedding spaces separate unless their compatibility has been established. Validate that a migrated memory remains retrievable by the intended agent and invisible to unauthorized tenants. Maintain provenance, and avoid embedding private data in logs used to monitor migration. LangGraph and Celery implementation In LangGraph , separate durable shared memory schemas from graph checkpoint schemas. A memory migration can affect future retrieval even when a paused graph's checkpoint still loads correctly. Validate both interactions where needed. In Celery , migration jobs should be retry safe, scoped to batches, and instrumented with record IDs and progress markers. Old queue messages can remain active after deployment; use compatible readers or version aware routing until they drain. Shared memory vs. workflow state Workflow state represents one run's progress, branches, and intermediate artifacts. Shared memory can outlive a run and be read or updated by many agents and workflows. Its migration must therefore account for long lived readers, retrieval indexes, concurrency, provenance, and tenant isolation. In short Migrate shared memory with versioned records, compatible readers, safe backfills, semantic validation, and explicit rollback planning. Correctness means agents retrieve the right meaning and access scope—not just that a new schema accepts the data.