How do you autoscale memory-compaction workers in a multi-agent system?

Autoscaling memory-compaction workers means adjusting the number of workers that build validated summaries as demand rises or falls, without starving other agents, exceeding model-provider limits, or allowing stale summaries to overwrite newer state. Autoscaling handles *capacity*; it does not replace sound scheduling, quality checks, or revision-safe activation.

Beginner-friendly example

A LangGraph workflow has a requirements agent, test generator, security reviewer, and publisher. Celery workers handle background compaction. During a quiet period, two workers keep up. After a large API-spec upload, 120 workflows request summaries within minutes, and the queue grows.

The platform can temporarily increase compaction-worker replicas, then reduce them as the queue drains. But if all replicas call the same LLM provider, launching more workers may simply produce rate-limit errors.

Measure incoming demand → Estimate capacity → Check external limits → Scale bounded workers → Preserve safe retries → Evaluate backlog and workflow latency → Scale down gradually

Numerical example: sizing workers

Suppose:

• New compaction jobs arrive at 24 jobs per minute.
• One worker has an average job execution time of 10 seconds.
• The team aims to keep utilization at or below 75% to leave headroom for variability.

One worker's theoretical service capacity is 60 / 10 = 6 jobs per minute. At 75% target utilization, its planned sustainable capacity is 6 × 0.75 = 4.5 jobs per minute.

Required workers ≈ ceil(24 / 4.5) = 6.

With only 3 workers, planned sustainable throughput is 3 × 4.5 = 13.5 jobs/minute; at the stated arrival rate, backlog would grow by approximately 10.5 jobs/minute, assuming the workload and service time remain steady. With 6 workers, planned capacity is 27 jobs/minute, leaving modest headroom.

However, if an external LLM API permits only 18 compaction calls per minute for this workload, six workers cannot sustainably complete 24 calls/minute. The true bottleneck is the provider limit. The system must reduce demand through deduplication, increase permitted quota, use an approved alternative model, or tolerate a queue. More workers alone cannot fix that ceiling.

These are simplified planning estimates. Production queueing requires checking service-time variability, retries, token-based limits, bursts, and end-to-end latency, not only averages.

What metrics should drive autoscaling?

1. Queue depth and oldest job age: Queue length shows work waiting; oldest age helps detect workflows approaching a context deadline.
2. Arrival and completion rates: Measure unique valid compaction requests per minute versus successfully validated summaries per minute—not just attempted tasks.
3. Queue latency and p95 execution time: Long jobs and variable model response times change required capacity.
4. Workers' active concurrency: Track jobs executing, waiting for network I/O, and worker saturation; CPU utilization alone may be misleading for LLM API-bound work.
5. Rate-limit headroom: Monitor remaining request/token quota, 429 errors, and retry-after signals; do not scale above a shared upstream cap.
6. Workflow impact: Track p95 end-to-end latency, compaction-blocked handoffs, context-overflow events, fallback retrieval, and successful test-suite publication.
7. Costs and quality: Watch compaction spend per accepted workflow, duplicate work, stale-candidate rejection, and summary-validation failure rates.

A scaling signal can combine queue pressure and deadline risk with a hard maximum based on vendor quota and cost policy.

Step-by-step production design

1. Isolate the work: Give compaction its own Celery queue and a controlled worker pool so background summaries cannot consume all workers needed for security reviews or publications.
2. Deduplicate requests: Coalesce tasks keyed by tenant + workflow + source revision + role; autoscale on *unique useful work* rather than a flood of duplicate attempts.
3. Choose the execution model: For I/O-bound API calls, modest worker concurrency may be effective; for CPU-heavy local summaries, size to actual CPU and memory constraints.
4. Define minimum and maximum replicas: Keep a small warm capacity where useful, and cap scaling according to provider quotas, spend limits, and infrastructure capacity.
5. Set target capacity: Estimate service rate per worker from observed p95 workload and utilization goals; recalibrate as source document sizes change.
6. Scale up on sustained pressure: Use queued-work age, arrival-rate pressure, and pending deadlines; react quickly enough to protect downstream context windows.
7. Scale down conservatively: Use a cooldown or stabilization period and avoid terminating workers that own important in-flight attempts. Support graceful shutdown, acknowledgments, and lease or visibility-timeout behavior appropriate to the queue.
8. Make redelivery harmless: Use stable task/summary IDs and conditional activation. Two workers must never create two conflicting active summaries for the same revision.
9. Provide overload behavior: If no validated summary is ready, the next agent may use a verified older summary with a complete event tail, retrieve targeted evidence, or block safely when the prompt would overflow.
10. Test before production: Simulate bursts, quota exhaustion, worker crashes, long-running jobs, stale revisions, and cross-tenant contention.

Queue-based autoscaling example

In Kubernetes, a queue-driven autoscaler such as KEDA can scale Celery worker deployments based on supported broker metrics. A native Kubernetes HPA can also use custom or external metrics when correctly configured.

For illustration, if there are 90 queued unique jobs and the desired pending backlog is 15 per worker, a simple queue-pressure estimate would request ceil(90 / 15) = 6 replicas, subject to configured minima/maxima. This heuristic is not a guarantee of a specific wait time: queued age, service time, incoming work, and provider limits determine actual latency.

Important: Queue length alone can be misleading when workers prefetch and reserve tasks. Consider ready, reserved, and active jobs, plus oldest pending age. Autoscaling based on only broker-visible messages may underestimate work already assigned to workers.

Prevent scaling from amplifying failures

Imagine the summarization provider starts returning 429 errors. New workers increase request pressure, causing more retries and even longer queues. A robust controller should distinguish capacity shortage from upstream throttling.

• Enforce a global request/token rate limit shared across workers.
• Honor retry-after responses and use bounded exponential backoff with jitter.
• Reduce or pause concurrency when failures indicate provider saturation.
• Use per-tenant quotas and fair scheduling to stop one customer from exhausting all resources.
• Alert if successful summary throughput drops while worker count rises.

LangGraph and Celery considerations

In LangGraph, queue compaction at a safe node boundary and persist the source revision and required consumer. A downstream model node must confirm that its selected summary is validated and fresh enough for the task. A graph checkpoint does not itself autoscale Celery workers or guarantee completion of a queued summary.

In Celery, workers can run on separately scaled deployments subscribed to a dedicated compaction queue. Configure task acknowledgment, prefetch, soft/hard time limits, and graceful termination with the workload's delivery semantics in mind. Autoscaling can cause redelivery during shutdown; therefore summary generation and activation must be idempotent and revision-checked.

How should success be evaluated?

Compare periods with and without autoscaling, controlling for workflow mix and load. Measure p95 compaction queue wait, oldest-job age, blocked agent steps, completed summaries per minute, overall workflow p95 latency, total compute/model cost, and validation quality. Include workload bursts and rate-limit incidents; do not claim a speedup merely because more workers were created.

Difference from earlier lessons

• Q200: Distributed rate limits across AI agents.
• Q230: When a workflow triggers compaction.
• Q231: Concurrency-safe summary publication.
• Q236: Cost and latency impact of compaction.
• Q237: Prioritization and queue scheduling of compaction jobs.
• This lesson: Dynamically determines how many compaction workers should run based on load, queue age, provider quotas, and safe scale-in behavior.

In short

Autoscale based on useful backlog and completion capacity—not CPU or queue length alone. Bound replicas by provider quotas and spend, preserve critical-agent capacity, and enforce idempotent, version-safe compaction so elasticity never sacrifices correctness.

Further reading

• Celery: Optimizing worker behavior
• KEDA: Scalers
• Kubernetes: Horizontal Pod Autoscaling