How do you implement distributed rate limiting across multiple AI agents?

Distributed rate limiting means enforcing one shared usage budget across all agents and workers, rather than letting each worker independently use the full quota. It is especially important when many agents call the same LLM provider, vector database, search API, or other rate-limited service.

Beginner-friendly example: shared LLM quota

Imagine three AI agents generating API tests. They all use one provider account capped at 120 requests per minute.

If each agent independently assumes it can send 120 requests per minute, combined demand could reach 360 requests per minute, or three times the shared limit.

Instead, all three request permission from the same rate limiter before calling the provider.

Agent requests permit → Shared limiter checks budget → Admit or delay → Call provider → Record outcome

Numerical example: token bucket

Suppose the shared limiter permits an average of 2 requests per second, with a bucket capacity of 4 requests.

1. Initially, the bucket has 4 tokens.
2. Agents A, B, and C simultaneously request one token each. All three are admitted; 1 token remains.
3. Two more requests arrive immediately. One is admitted; the other must wait or receive a retryable limit response.
4. After 1 second, approximately 2 tokens are replenished, subject to the configured cap of 4.

The bucket allows short bursts while maintaining a controlled long-term request rate. Real providers can also apply token-per-minute limits, concurrent-request caps, and separate quotas per model; one request token is not equivalent to one LLM output token.

How do you implement shared limits?

1. Identify quota scope: Determine whether limits apply per organization, provider key, model, tenant, agent, or endpoint.
2. Use a shared coordinator: Implement an atomic token-bucket or sliding-window counter with a suitable centralized store or gateway.
3. Acquire permits before external calls: Every agent and Celery worker must use the same enforcement path.
4. Track multiple dimensions: Enforce requests per minute, token budgets, concurrent calls, and daily cost ceilings separately where needed.
5. Reserve or estimate usage safely: For token quotas, use conservative estimates and reconcile with actual reported usage.
6. Handle denial predictably: Queue within deadline and capacity limits, postpone work, or return an explicit rate-limited status.
7. Respond to provider signals: Honor documented retry-after guidance and throttle following provider-side rate-limit responses.
8. Measure fairness: Prevent one tenant or workflow from monopolizing all permits.

Example implementation sketch

A shared limiter might maintain a record such as:

JSON
{
  "scope": "provider-account:model-A",
  "requests_per_minute": 120,
  "max_concurrent_requests": 10,
  "bucket_capacity": 4
}

This is an illustrative policy record, not a complete token-bucket configuration. Atomic updates, refill timing, and expiry rules are required in a real implementation.

What if the shared limiter is unavailable?

Choose a deliberate fail-closed policy for strict provider or billing limits, or a tightly bounded fallback budget when the business accepts the risk. Do not let every worker assume the full global quota. Monitor limiter errors and avoid synchronized retries that create a thundering herd.

LangGraph and Celery integration

In LangGraph, agent nodes can acquire shared permits before LLM or tool calls. A workflow deadline determines whether to wait, switch to an approved alternative, or report a deferred task.

With Celery, tasks running on many worker processes or machines need a shared coordinator; a per-process semaphore does not enforce an account-wide request budget. Broker routing and worker concurrency can help shape load but cannot by themselves enforce a provider's global quota.

Distributed rate limiting vs. backpressure

Rate limiting enforces an explicit usage allowance across requesters. Backpressure slows producers when downstream capacity is overwhelmed. They often work together: the shared limiter controls provider requests while bounded queues prevent work from piling up indefinitely.

In short

Enforce one atomic, shared quota across every agent using the same constrained dependency. Combine request, token, and concurrency budgets with fair scheduling and clear responses when capacity is exhausted.