AI Agents
An AI agent is a system that uses a model to choose actions in an environment. Agents often combine reasoning, memory, tools, retrieval, and feedback so they can complete multi-step tasks.
How do you prevent cascading failures in multi-agent systems?
A cascading failure happens when trouble in one agent or dependency spreads through the rest of a multi agent workflow. For example, an unavailable model service can trigger retries, fill shared queues, exhaust worker slots, and eventually delay agents that do not use that model directly. The goal is not simply to recover one failed agent. It is to stop the chain reaction before it takes down otherwise healthy parts of the system. Beginner friendly example: API test generation A platform has three stages: Requirements agent extracts API specifications. Test generation agent calls an external LLM. Review agent validates generated suites. The external LLM slows down. The test generator begins timing out, so the orchestrator retries every request. Those retries occupy workers and queue connections used by the review agent. Soon, review tasks also time out—even though the reviewer itself has no defect. Slow dependency → More waiting tasks → Retry amplification → Shared resource exhaustion → Unrelated agents fail Numerical example: retry amplification Suppose 20 requests per minute arrive and each failed generation attempt is retried twice . Without retries: up to 20 attempts per minute . With two retries per request: up to 20 × 3 = 60 attempts per minute , if all attempts fail and retry timing sustains that rate. The retry policy has potentially tripled demand on an already struggling dependency. Multiple agent layers retrying independently can amplify traffic even further. This is why retries should have a shared budget rather than each layer blindly repeating the same operation. How do you prevent cascading failures? 1. Map failure propagation paths: Identify which agents share models, databases, tools, queues, and worker pools. Draw the dependency chain, not just the agent graph. 2. Bound work at every boundary: Set end to end deadlines, maximum in flight tasks, queue limits, and per dependency concurrency budgets. 3. Use a coordinated retry budget: Limit retries across the whole request, with exponential backoff and jitter. Do not retry permanent validation or authorization failures. 4. Stop calling unhealthy dependencies: Apply circuit breakers to repeated dependency failures so traffic does not keep intensifying an outage. 5. Separate failure domains: Use bulkheads such as independent worker pools or quotas so one saturated specialist cannot consume all available resources. 6. Propagate structured failure status: Tell upstream nodes whether an operation is failed, retryable, pending, or blocked. Never disguise a timeout as a successful response. 7. Degrade only where safe: Continue independent work, but stop approval if a mandatory security or correctness stage is incomplete. 8. Recover gradually: Allow limited probes and controlled traffic ramp up instead of releasing the entire backlog immediately. What makes this different from other resilience patterns? Backpressure controls how much work enters an overloaded stage. A circuit breaker temporarily blocks calls to one failing dependency. Fault isolation separates resources and state to contain damage. Cascading failure prevention combines these controls across the full dependency graph, with particular attention to retry amplification, shared bottlenecks, and propagation between stages. LangGraph and Celery example In LangGraph , store a single workflow deadline and a retry budget in trusted orchestration state. A recovery branch can inspect the failing dependency, stop repeated delegation, and leave independent branches running when policy permits. With Celery , monitor queues, retries, active workers, and provider rate limits across all relevant worker pools. Separate critical work and set limits that account for shared external services. A larger worker pool does not help if every worker is retrying the same rate limited provider. Alert on simultaneous queue growth, elevated retries, increased timeouts, and falling successful throughput . Together, these signals can reveal a cascading incident earlier than isolated error counters. In short Prevent cascading failures by controlling retries, deadlines, queue growth, shared resource consumption, and failure propagation across the entire multi agent workflow. The system should contain the original problem, preserve healthy work, and recover in stages.
How do you manage agent communication during network partitions?
A network partition happens when some agents or services cannot communicate with others, even though their processes may still be running. A multi agent system must handle this uncertainty without duplicating important actions or accepting conflicting updates. Beginner friendly example: API test generation A coordinator delegates security test generation to Agent A, but the network connection breaks after A receives the job. The coordinator cannot reach A and does not know whether work started. Agent A may still be generating test cases. Sending the same job to Agent B immediately could create two competing outputs. The correct response is not simply to retry every request. Preserve the logical task ID, mark the outcome unknown , and follow a recovery policy. Detect lost connectivity → Preserve task state → Restrict risky writes → Wait or reassign safely → Reconcile after reconnection Simple numerical example Imagine three workers share a job system: Two workers can reach the authoritative database. One worker loses database access but can still call an LLM. All three were previously capable of generating test cases. If the isolated worker continues publishing local results while the other two process reassigned work, the system may end up with duplicate suites. Instead, the isolated worker must not commit authoritative completion without validating its current ownership. The two connected workers may continue processing tasks through the authoritative store, but only if they satisfy the system's normal lease, fencing, and authorization rules. Two out of three reachable workers is not automatically a safe quorum ; a quorum matters only when supported by the actual coordination protocol. How should communication behave during a partition? 1. Use finite deadlines and health signals: Treat silence as uncertainty, not proof that the remote agent is dead. 2. Persist messages and task progress: Prefer durable queues and workflow checkpoints rather than memory only state. 3. Assign stable logical task IDs: Recognize retries and redeliveries as attempts at the same work. 4. Protect ownership: Coordinate claims through an authoritative store and reject writes from obsolete owners using fencing or equivalent safeguards. 5. Separate safe from unsafe work: Agents may continue independent computations, but operations requiring global coordination or irreversible side effects should pause if authority cannot be established. 6. Bound retries and queues: Use backoff, deadlines, and backpressure to avoid a reconnect storm. 7. Reconcile after healing: Compare task status, versions, and output artifacts before accepting late responses or retrying missing work. Consistency vs. availability During a partition, a distributed system cannot always provide both fully consistent shared state and uninterrupted writes. For critical actions such as approving a security review, choose a conservative policy that requires authoritative state. For temporary draft generation, local work may be acceptable if the result remains explicitly provisional. This tradeoff depends on the system's architecture, not a universal rule that every agent must stop. LangGraph and Celery considerations In LangGraph , checkpoints can preserve workflow progress, but coordination among disconnected workers still needs an authoritative state and recovery design. Store request IDs and task versions so resumed branches can reject stale updates. With Celery , broker connectivity loss, worker availability, and result backend connectivity are distinct failures. Delivery and redelivery behavior depend on broker and task acknowledgment settings. Test actual failure scenarios; do not assume that a timeout cancels remote execution. Common mistakes Treating missing heartbeats as proof an agent has stopped. Allowing two disconnected coordinators to approve conflicting results. Assuming a durable queue guarantees exactly once execution. Retrying so aggressively that the recovered system is immediately overloaded. Automatically trusting results generated offline without version or permission checks. In short Handle network partitions by preserving identity and state, limiting unsafe actions during uncertainty, and reconciling work after connectivity returns. Reliability means avoiding both lost work and contradictory outcomes—not merely reconnecting as quickly as possible.
How do you recover agent workflows after a network partition?
Recovering a multi agent workflow after a network partition means reconciling what each agent actually completed while connectivity was lost, then resuming only work that is still valid and unfinished. Reconnection alone is not recovery: agents may hold conflicting views of task ownership, versions, and results. Beginner friendly example: API test generation A coordinator assigns the security test agent a job for POST /payments . The connection breaks after the agent starts. During the partition: The coordinator marks the result unknown and prepares a replacement attempt. The original security agent finishes and saves a local draft. Another authorized agent generates a second draft after obtaining a new task claim. When communication returns, both drafts exist , but only one may be eligible to become the official output. Reconnect → Read authoritative task state → Collect late outcomes → Compare ownership and versions → Accept or reject results → Resume missing work Numerical example: deciding which result wins Suppose logical task payment security 42 has these attempts: Attempt A: Ownership token 11 ; completed locally during the partition. Attempt B: Ownership token 12 ; obtained a valid new claim and saved its result. If the authoritative result store enforces fencing tokens, token 12 supersedes token 11 . The workflow must reject A's late write even if its generated content looks useful. If no effective fencing was enforced, first investigate whether any conflicting side effects were already committed; do not assume a higher number alone repairs them. The accepted result must still pass schema, authorization, and quality checks. A safe reconciliation procedure 1. Re establish trusted connectivity: Authenticate agents and verify access to the authoritative database, broker, and artifact store. 2. Freeze conflicting writes when needed: Avoid automatically replaying all offline actions while ownership is uncertain. 3. Read durable state: Compare workflow checkpoints, task statuses, current owners, versions, and recorded outputs. 4. Collect and classify late messages: Identify completed, failed, pending, duplicated, and stale attempts using stable task IDs. 5. Resolve conflicts: Apply documented version or fencing rules; reconcile external side effects separately. 6. Resume only missing operations: Requeue unfinished tasks with bounded retries and current permissions; do not blindly rerun completed work. 7. Validate dependencies: Ensure resumed stages consume the accepted version of upstream artifacts. 8. Restore traffic gradually: Drain accumulated queues without creating a retry storm; record the recovery decision for audit. What if two agents changed the same state? For shared authoritative state, prefer transactional version checks or a single accepted owner. For independent, mergeable outputs—for example, test cases covering distinct endpoints—the system may combine validated artifacts if their versions and permissions are compatible. Not every conflict should be merged. Two competing security approvals, account changes, or external writes may require manual investigation. LangGraph and Celery example LangGraph: Persist the workflow's logical task IDs and accepted artifact references in checkpoints. After reconnection, compare checkpoint state with the authoritative task/result store, then route only unresolved branches through a recovery node. Celery: Broker redelivery and worker acknowledgments may have occurred during the outage. Query durable task status and artifacts before requeueing; worker or result backend reconnection does not prove that a previously timed out task never ran. Shared claim records and fencing remain important. Difference from earlier concepts Resuming a failed workflow is the general process of continuing from saved progress. Managing a network partition focuses on safe behavior while components cannot communicate. Post partition recovery focuses on reconciling divergent attempts, late responses, conflicting versions, and authoritative results after connectivity returns . In short Reconcile first, resume second. Trust durable task identity and ownership records, reject stale outcomes, verify external effects, and restart only the work that is truly incomplete.
How do you maintain consistency of shared state across multiple agents?
Shared state consistency means multiple agents see and update a common workflow record without silently overwriting one another's work. Agents may execute in parallel, but updates to authoritative task status, accepted artifacts, and approvals must follow explicit concurrency rules. Beginner friendly example: API test generation Two agents review the same payment endpoint: Agent A adds authorization test cases. Agent B adds invalid input test cases. Both read test suite version 5 . Agent A saves an update, producing version 6 . If Agent B then writes its entire outdated version 5 copy, it may erase Agent A's changes. This is a lost update . Read version → Prepare changes → Check expected version → Commit or resolve conflict → Publish new version Step by step numerical example The shared record initially contains 20 test cases , at version 5 . 1. Agent A reads version 5 and proposes 3 new tests . 2. Agent B also reads version 5 and proposes 2 new tests . 3. Agent A commits, producing 23 tests at version 6 . 4. Agent B's attempt to replace version 5 is rejected because the current version is 6. 5. Agent B reads version 6, checks that its two tests are distinct, then commits 25 tests at version 7 . Without the version check, Agent B could overwrite A's three tests and leave only 22. The numbers assume all additions are valid and unique. Approaches to preserving consistency 1. Single authoritative store: Choose where the accepted workflow status and output version live; do not treat every agent's local memory as equally authoritative. 2. Optimistic concurrency: Require an expected version on updates, and re read or merge when it changes. 3. Transactions: Make logically related changes atomic—for example, save an accepted result and mark its task complete together. 4. Scoped ownership: Give one agent responsibility for approving final shared state while specialists produce separate draft artifacts. 5. Append only events or artifacts: Record independent contributions separately, then combine them through a validated aggregation step. 6. Conflict policies: Distinguish safely mergeable additions from conflicting approvals, replacements, or deletions. 7. Version aware reads: Ensure downstream agents consume the correct accepted artifact version rather than a stale cache. Simple PostgreSQL example An optimistic update can check the version it read: A returned row means the update succeeded. Zero rows means the record was not updated; determine whether the suite is missing or another writer changed its version. The agent must not assume success or repeatedly retry the same stale payload. How should parallel agents share data? Use immutable or versioned outputs for independent branches, then let a controlled aggregator assemble the final result. This often reduces contention compared with letting every agent edit one large JSON document. For example, keep separate references for security tests and validation tests; when both complete, the aggregator checks source specification versions, deduplicates cases, and records one approved suite. Shared state is not the same as shared memory: memory can provide useful context, while authoritative workflow state needs clear ownership, durability, and concurrency controls. LangGraph and Celery implementation In LangGraph , define how state updates from parallel branches are combined and avoid assuming all concurrent writes automatically merge safely. Reducers can combine compatible updates, but reducers alone cannot prevent races on an external database or shared artifact store. With Celery , several workers may run simultaneously or retry tasks. Use database transactions and version checks for shared records; Celery task execution order is not a reliable serialization mechanism. In short Maintain shared state consistency with authoritative storage, version checks, atomic transactions, and explicit merge rules. Parallel agents can work independently, but committing accepted results must remain controlled and auditable.
What is optimistic vs. pessimistic locking in multi-agent systems?
Optimistic locking and pessimistic locking are two ways to stop concurrent agents from making conflicting changes to shared data. The key difference is when conflicts are handled: optimistic locking checks for them at commit time, while pessimistic locking reserves the data before making a change. Beginner friendly example: API test generation agents Two agents need to update the same test suite: Agent A adds authorization tests. Agent B adds boundary value tests. Both can read the suite, but their updates must not silently erase each other. 1. Optimistic locking: assume conflicts are uncommon Each record has a version number. Agents read the current version, prepare changes, and submit an update only if that version is still current. Suppose the suite starts at version 4 with 10 tests . 1. A and B both read version 4. 2. A adds 3 tests and commits, producing version 5 with 13 tests . 3. B tries to commit its version 4 copy with 2 additional tests. The version check rejects the stale update. 4. B reloads version 5, merges its distinct additions, and commits version 6 with 15 tests . An illustrative PostgreSQL update is: If no row is returned, the caller must check whether another writer changed the record or the record does not exist. Do not retry the same stale content without reconciling it. 2. Pessimistic locking: reserve the record first The agent acquires a database lock before reading and changing the shared record. Another transaction trying to acquire an incompatible lock must wait or fail. For example, in PostgreSQL: The example assumes a transactional database connection and appropriate permissions. Do not keep this database transaction open while waiting for an LLM response : long held locks block other writers and can increase contention. When should you choose each? Optimistic locking: Best when conflicts are relatively rare, changes can be recalculated, and long model calls happen outside the transaction. Failed commits require explicit conflict handling. Pessimistic locking: Useful when short critical sections have high contention and conflicts would be expensive. It can increase waiting time and cause deadlocks if locks are acquired inconsistently. Separate immutable artifacts: Often even better for independent agent work. Each specialist writes its own result, and a controlled aggregator merges them afterward. Are database locks the same as distributed leases? No. PostgreSQL row locks protect changes during a database transaction . A distributed lease grants temporary ownership across workers and typically needs expiry and fencing rules. A short row lock cannot guard an agent that spends minutes generating text outside that transaction. For a long running task, use stable task ownership and version checks around final writes—not an open database lock spanning the whole LLM operation. LangGraph and Celery implementation In LangGraph , parallel branches can produce separate results, but writes to a shared external record still require application level concurrency control. State reducers alone do not lock a database row. With Celery , multiple workers may attempt to update the same suite concurrently. Use the database's version check or brief transaction; ordering of Celery task completion does not serialize shared writes. In short Optimistic locking detects conflicting writes when committing; pessimistic locking blocks conflicting access beforehand. For AI agents with long running model calls, optimistic versioning plus short transactions and separately stored artifacts is often a practical starting point.
What are distributed locks and leases in multi-agent systems?
A distributed lock is a coordination mechanism that helps multiple agents agree which worker may perform a particular operation. A lease is time limited ownership that expires unless renewed. These mechanisms matter when agents run on different machines and cannot safely rely on a local in memory lock. Beginner friendly example A multi agent API testing platform has two security test workers. Both are ready to generate the official security suite for the same payment API. Agent A obtains a lease for task payment security 42 for 30 seconds . Agent B attempts to acquire the same lease but is denied while A owns it. A completes the task, persists its result, and releases ownership. If A crashes, its lease can expire so another eligible worker can take over. Claim lease → Work under ownership → Renew if needed → Validate ownership before committing → Release or expire This prevents unnecessary competition, but a lease alone does not guarantee that expired workers stop running. Numerical example: why expiry is not enough Suppose: 1. At 00:00 , Agent A receives a 30 second lease with fencing token 7 . 2. At 00:25 , Agent A pauses unexpectedly for 20 seconds . 3. At 00:30 , A's lease expires. Agent B obtains a new lease with fencing token 8 . 4. At 00:45 , Agent A resumes and attempts to publish its result. Even though A was the original owner, its lease has expired . A must not overwrite B's accepted result. A downstream database or service that checks fencing tokens should reject A's older token 7 . What should a lease record contain? Resource or task key: What work is being protected. Owner ID: Which agent or worker currently holds the claim. Expiry time: When the claim must be renewed. Fencing token: A monotonically increasing ownership generation. Optional metadata: Workflow ID, tenant scope, and audit timestamps. Store ownership in a shared, trustworthy coordination system. Changes to owner, expiry, and fencing generation must be atomic . How does renewal work? A worker periodically renews the lease while working. Renewal must verify that it is still the current owner; a worker must not extend somebody else's lease. If renewal fails or ownership is lost, the worker should stop initiating protected side effects. For a 30 second lease, a worker might attempt renewal every 10 seconds . That timing is illustrative: choose durations based on scheduling pauses, network delays, and failure detection requirements. A lease is not a reliable “kill switch.” An isolated or paused worker may continue running after expiry. Fencing enforcement at the protected resource is what prevents stale ownership from committing authoritative changes. Distributed locks vs. database row locks Database row lock: Usually lasts for the duration of a transaction. Use it for short, atomic changes. Distributed lease: Grants temporary task ownership across machines and survives beyond a single database transaction. Optimistic version check: Detects concurrent changes when committing, without reserving a long running operation. Avoid holding a database transaction open during a long LLM generation call. Use a short transaction to claim/renew a lease, perform the work outside the transaction, then validate ownership on commit. LangGraph and Celery integration A LangGraph coordinator can associate each specialist task with a stable task key and record the accepted artifact separately from the agent's draft. For Celery workers, a shared lease mechanism can coordinate tasks that might be retried or redelivered on different workers. Do not assume Celery task IDs or LangGraph checkpoints are distributed locks. Neither alone guarantees exclusive task ownership across running processes. Operational safeguards Make acquisition, renewal, and release atomic and owner checked. Use a trustworthy source of time for expiry decisions and avoid unsafe assumptions about clock synchronization. Use fencing tokens where stale workers could affect external state. Monitor expired leases, failed renewals, blocked claims, and stale write attempts. Retain authorization and tenant scoping rules; a lock must not grant access rights. In short Distributed locks coordinate access; leases bound ownership in time; fencing tokens protect against workers that resume after losing ownership. Use them together when several agents may compete to perform the same authoritative operation.
How do you handle deadlocks between multiple agents?
A deadlock occurs when two or more agents each wait for a resource or action held by another, so none can make progress. In a multi agent workflow, this can happen with database locks, distributed leases, approval handoffs, or circular task dependencies. Beginner friendly example: API test generation Two specialist agents need exclusive access to two shared resources: Agent A acquires lock Requirements and then waits for TestSuite . Agent B acquires lock TestSuite and then waits for Requirements . Neither releases its first lock because each is waiting for the other. The workflow stops moving even though both agents are alive. A holds Requirements → A waits for TestSuite → B holds TestSuite → B waits for Requirements → Deadlock Numerical example Suppose both agents have a 10 second lock wait timeout: 1. At 0 seconds , A holds Requirements while B holds TestSuite. 2. At 1 second , A requests TestSuite and B requests Requirements. 3. Neither can proceed. At about 11 seconds , the wait timeouts can break the cycle if the application aborts an attempt and releases its held resources. 4. The aborted task can retry after a bounded, jittered delay, acquiring resources in a fixed order. A timeout alone is not sufficient if the task keeps holding other resources or immediately retries in the same conflicting order. How can you prevent deadlocks? 1. Use a global lock order: Every agent acquires shared resources in the same order, such as Requirements before TestSuite. 2. Keep critical sections short: Do not hold database row locks while waiting for LLM responses, human approval, or slow network calls. 3. Avoid nested locks when practical: Produce separate immutable artifacts and combine them in a short controlled transaction. 4. Set lock wait timeouts: Fail or reschedule work instead of waiting indefinitely. 5. Detect cycles: For application level dependencies, maintain a wait for graph and look for cycles among tasks or resources. 6. Release ownership on failure: Roll back failed transactions, release owned locks safely, and use expiring leases for longer work. 7. Retry carefully: Use bounded retries with backoff and jitter; change the contention pattern instead of repeating a known deadlock. How do you detect and recover? A database such as PostgreSQL can detect certain transaction deadlocks and abort one transaction. The application must handle that error, roll back the failed unit of work, and decide whether a safe retry is appropriate. For logical agent deadlocks , such as Agent A waiting for B's approval while B waits for A's confirmation, a database cannot detect the cycle automatically. An orchestrator should track dependencies, enforce deadlines, and escalate or break circular handoffs according to a defined policy. LangGraph and Celery example In LangGraph , define task dependencies so an approval or handoff does not create an unintended cycle. Instrument waiting states and use explicit timeout/recovery branches. A graph cycle is not automatically a deadlock—loops can be intentional when they have progress and termination conditions. With Celery , avoid workers synchronously waiting for results of other tasks when the same limited worker pool must execute those tasks. For example, if all 4 worker slots are blocked waiting for subtasks that require those same 4 slots, no subtask can start. Prefer asynchronous composition, callbacks, or dedicated capacity instead of occupying workers while waiting. Deadlock vs. slow processing Deadlock: Tasks are blocked by a cycle of dependencies or resource waits. Starvation: A task waits indefinitely because others repeatedly receive priority or resources. Slow processing: Work still progresses, but takes longer than expected. These problems have different fixes; adding more agents does not necessarily resolve a deadlock. In short Prevent deadlocks through consistent resource ordering, short transactions, nonblocking orchestration, and bounded waits. Detect circular dependencies early, abort or reschedule safely, and never assume that healthy agents guarantee a progressing workflow.
How do you prevent starvation in multi-agent task scheduling?
Starvation occurs when a task waits indefinitely because other tasks repeatedly receive priority or access to limited agent capacity. It differs from a deadlock: work continues elsewhere, but the neglected task never gets a fair opportunity to run. Beginner friendly example: API test generation A multi agent testing service accepts two types of work: Urgent security tests: High priority. Routine regression tests: Lower priority. The system has a single available reviewer. New urgent jobs keep arriving, so the scheduler always selects them first. A regression job may remain queued for hours even though the reviewer is constantly working. Accept tasks → Track waiting time → Apply fairness policy → Select eligible task → Execute → Monitor queue age Numerical example: priority aging Suppose jobs have a base priority of 10 for urgent requests and 2 for routine requests, where higher is better. A scheduler increases each pending job's effective priority by 1 point every 5 minutes : Effective priority = base priority + floor(waiting minutes / 5) A newly arrived urgent job scores 10 . A routine job waiting 20 minutes scores 2 + 4 = 6 . After 40 minutes , that routine job scores 2 + 8 = 10 . After 45 minutes , it scores 11 , overtaking a newly arrived urgent job. This is aging : a task gradually gains priority as it waits. The example illustrates the mechanism; it does not guarantee a wait time bound when urgent jobs are already queued or capacity is insufficient. Production scheduling must define tie breaking, eligibility, and service level limits. Strategies to prevent starvation 1. Priority aging: Increase the priority of waiting tasks gradually. 2. Weighted fair scheduling: Allocate a defined share of worker time to different classes or tenants. 3. Reserved capacity: Keep some worker slots for lower priority work when service requirements justify it. 4. Maximum wait thresholds: Alert, escalate, or temporarily elevate tasks that exceed allowed queue age. 5. Round robin across eligible queues: Serve different job classes in rotation where task costs are reasonably comparable. 6. Per tenant quotas: Prevent one customer or workflow from monopolizing all available concurrency. 7. Cost aware fairness: Account for task duration or estimated resource use; fair task counts do not always mean fair compute consumption. Fairness is not the same as equal priority A production security incident may legitimately take precedence over a routine report. Starvation prevention does not require identical service levels. Instead, define minimum service guarantees and explicit exceptions for critical work. Measure oldest queued task age, waiting time distributions, tenant shares, throughput, and missed deadlines —not just average queue length. LangGraph and Celery considerations In LangGraph , a coordinator may tag jobs with priority, deadline, workflow ID, and tenant, then use a scheduler to determine when eligible branches are dispatched. A graph's execution order alone does not provide cross workflow fairness. With Celery , queues, routing, worker concurrency, and broker specific priority behavior influence scheduling. Celery message priority is not a complete fairness policy. A separate scheduler, dedicated queues, or admission controller may be needed to enforce aging or weighted shares. Test against the actual broker, prefetch configuration, and workload distribution. Starvation vs. related problems Deadlock: Multiple agents or tasks block one another in a dependency cycle. Starvation: Other work progresses, but one task receives no service. Backpressure: The system limits incoming work when capacity is insufficient. Load balancing: Accepted work is distributed among eligible agents. These concepts complement one another but require different remedies. In short Prevent starvation by guaranteeing meaningful access to capacity for tasks that would otherwise wait indefinitely. Combine priority aging, fair shares, bounded waiting policies, and monitoring without ignoring genuinely urgent work.
How do you detect and handle livelocks in multi-agent systems?
A livelock happens when multiple agents remain active—retrying, handing tasks back and forth, or changing plans—but make no useful progress. Unlike a deadlock, where work is blocked, livelocked agents keep doing work that fails to move the shared goal closer to completion. Beginner friendly example: API test generation agents A requirements agent and a test generator disagree about the expected input: Requirements agent: Sends a security requirement without an authentication scheme. Test generator: Rejects it and sends it back for clarification. Requirements agent: Resends the unchanged requirement. Test generator: Rejects it again. Both agents are responding, yet no test suite is produced. Observe repeated handoffs → Measure progress → Stop the cycle → Diagnose the missing dependency → Escalate or choose a bounded recovery action Simple numerical example Suppose a workflow allows at most 5 handoffs for a given requirement, and each handoff costs 2 seconds . After 5 handoffs , the system has spent 10 seconds . The count of validated requirements remains 0 . The coordinator marks the task as STALLED , records the repeated state, and requests a missing authentication scheme rather than starting the same exchange again. A useful signal is not merely that many messages occurred; it is that progress stayed unchanged while the same states or errors recurred. The thresholds are illustrative and should reflect task complexity. How can you detect a livelock? 1. Define measurable progress: Examples include validated fields, completed subtasks, accepted artifacts, or reduced unresolved errors. 2. Track state transitions: Record handoffs, retries, decisions, and whether they changed the useful task state. 3. Recognize repeated states: Detect cycles such as A → B → A with identical inputs, version numbers, or unresolved errors. 4. Set bounded budgets: Limit retries, handoffs, elapsed time, model calls, and token expenditure. 5. Alert on no progress: Trigger review when the system remains active but a progress metric is unchanged beyond a defined threshold. Do not confuse a legitimate multi step refinement loop with livelock. Revisions that genuinely improve an artifact or reduce validation errors may represent meaningful progress. How do you break a livelock? Change the recovery action: Request missing data, choose a different approved strategy, or escalate to a human instead of replaying the same message. Introduce asymmetric coordination: Let a designated supervisor make a final routing decision when peers repeatedly defer to one another. Use randomized backoff where relevant: Prevent agents contending for the same resource from repeatedly yielding in sync. Apply a termination policy: End with a clear STALLED or FAILED status and preserve usable intermediate results. Require evidence of improvement: Permit another attempt only when an input, plan, dependency, or version has materially changed. LangGraph and Celery considerations In LangGraph , track a structured progress marker and handoff count in graph state. Add a conditional route that stops repeated A↔B transitions when a budget or no progress threshold is reached. Graph recursion limits can bound execution but do not by themselves diagnose the reason for a livelock. In Celery , repeated retries or rapidly resubmitted tasks can produce a livelock like cycle while workers remain busy. Keep shared retry budgets and task state records in a durable store; avoid allowing each worker to reset the retry count independently. Deadlock vs. starvation vs. livelock Deadlock: Agents wait on each other and cannot proceed. Starvation: Other tasks progress, but one task repeatedly loses access to resources. Livelock: Agents continue taking actions, but those actions fail to advance the shared task. In short Detect livelocks by tracking useful progress, not just agent activity. Break repeating cycles with explicit budgets, changed strategies, supervisor decisions, and truthful stalled task reporting.
How do you implement distributed rate limiting across multiple AI agents?
Distributed rate limiting means enforcing one shared usage budget across all agents and workers, rather than letting each worker independently use the full quota. It is especially important when many agents call the same LLM provider, vector database, search API, or other rate limited service. Beginner friendly example: shared LLM quota Imagine three AI agents generating API tests. They all use one provider account capped at 120 requests per minute . If each agent independently assumes it can send 120 requests per minute, combined demand could reach 360 requests per minute , or three times the shared limit . Instead, all three request permission from the same rate limiter before calling the provider. Agent requests permit → Shared limiter checks budget → Admit or delay → Call provider → Record outcome Numerical example: token bucket Suppose the shared limiter permits an average of 2 requests per second , with a bucket capacity of 4 requests . 1. Initially, the bucket has 4 tokens . 2. Agents A, B, and C simultaneously request one token each. All three are admitted; 1 token remains . 3. Two more requests arrive immediately. One is admitted; the other must wait or receive a retryable limit response. 4. After 1 second , approximately 2 tokens are replenished, subject to the configured cap of 4. The bucket allows short bursts while maintaining a controlled long term request rate. Real providers can also apply token per minute limits, concurrent request caps, and separate quotas per model; one request token is not equivalent to one LLM output token. How do you implement shared limits? 1. Identify quota scope: Determine whether limits apply per organization, provider key, model, tenant, agent, or endpoint. 2. Use a shared coordinator: Implement an atomic token bucket or sliding window counter with a suitable centralized store or gateway. 3. Acquire permits before external calls: Every agent and Celery worker must use the same enforcement path. 4. Track multiple dimensions: Enforce requests per minute, token budgets, concurrent calls, and daily cost ceilings separately where needed. 5. Reserve or estimate usage safely: For token quotas, use conservative estimates and reconcile with actual reported usage. 6. Handle denial predictably: Queue within deadline and capacity limits, postpone work, or return an explicit rate limited status. 7. Respond to provider signals: Honor documented retry after guidance and throttle following provider side rate limit responses. 8. Measure fairness: Prevent one tenant or workflow from monopolizing all permits. Example implementation sketch A shared limiter might maintain a record such as: This is an illustrative policy record, not a complete token bucket configuration. Atomic updates, refill timing, and expiry rules are required in a real implementation. What if the shared limiter is unavailable? Choose a deliberate fail closed policy for strict provider or billing limits, or a tightly bounded fallback budget when the business accepts the risk. Do not let every worker assume the full global quota. Monitor limiter errors and avoid synchronized retries that create a thundering herd. LangGraph and Celery integration In LangGraph , agent nodes can acquire shared permits before LLM or tool calls. A workflow deadline determines whether to wait, switch to an approved alternative, or report a deferred task. With Celery , tasks running on many worker processes or machines need a shared coordinator; a per process semaphore does not enforce an account wide request budget. Broker routing and worker concurrency can help shape load but cannot by themselves enforce a provider's global quota. Distributed rate limiting vs. backpressure Rate limiting enforces an explicit usage allowance across requesters. Backpressure slows producers when downstream capacity is overwhelmed. They often work together: the shared limiter controls provider requests while bounded queues prevent work from piling up indefinitely. In short Enforce one atomic, shared quota across every agent using the same constrained dependency. Combine request, token, and concurrency budgets with fair scheduling and clear responses when capacity is exhausted.