How do you implement backpressure when memory-compaction demand exceeds worker capacity?

Backpressure in memory compaction means slowing, deferring, merging, or rejecting new compaction work when demand exceeds the capacity of workers or upstream language-model APIs. Without it, queued summaries grow stale, workers exhaust resources, and agents can wait longer for context than they would without compaction.

The goal is controlled degradation: preserve critical workflow decisions and source evidence while refusing to let an optional optimization overwhelm the system.

Beginner-friendly production incident

An API-test generation platform runs requirements, test-generation, security-review, and publication agents in LangGraph. Celery workers summarize old messages before expensive handoffs.

A newly uploaded specification triggers a burst of workflows. Hundreds of background compaction requests enter the queue, but an external summarization model can serve only a limited number per minute. Celery retries rate-limited calls, increasing the queue further. The security-review agents begin timing out while waiting for summaries.

The root cause is unbounded admission plus a retry feedback loop, not necessarily too few CPU cores. The solution is to bound useful compaction demand, isolate it from critical work, and let agents use a safe, verified alternative when a new summary is unavailable.

Measure overload → Stop admitting redundant work → Enforce queue and quota limits → Prioritize urgent jobs → Use safe fallback → Recover gradually

Step-by-step numerical example

Assume compaction requests arrive at 50 jobs per minute, while the complete system—including model-provider limits—can successfully finish only 30 per minute.

1. Net backlog growth = 50 − 30 = 20 jobs/minute, assuming no deduplication or retries.
2. After 5 minutes, the queue grows by about 100 jobs.
3. At a stable service rate of 30 jobs/minute, clearing those 100 already queued jobs would take at least 100 / 30 ≈ 3.33 minutes *if no new work arrived*. Under continued arrival of 50/minute, the backlog instead keeps growing.
4. Suppose admission control coalesces 15 duplicate or obsolete requests per minute and defers 10 nonurgent requests per minute. New admitted demand falls to 25 jobs/minute.
5. With a capacity of 30 jobs/minute, the backlog can now shrink by approximately 5 jobs/minute, provided the deferred requests do not immediately reenter the queue and retries are controlled.
6. A starting backlog of 100 therefore needs roughly 20 minutes to drain at the 5-job/minute net rate. If the team temporarily stops new admission, it could drain in about 3.33 minutes instead.

These are illustrative steady-state estimates. Actual latency depends on service-time variability, burstiness, reserved tasks, token quotas, priority classes, and failures.

How do you recognize compaction overload?

• Incoming unique jobs exceed validated completions for a sustained period.
• Queue depth, oldest queued job age, and p95 wait time rise.
• Summaries finish after their source revision is obsolete; stale-candidate rejection grows.
• Model-provider 429 responses, timeouts, and retry counts increase.
• Other agent tasks wait because compaction shares worker slots or API quotas.
• More workflows reach the next-model-call context limit before an acceptable summary is ready.

A long queue alone is not proof of an incident: compare expected completion deadlines, actual downstream blocking, and successful throughput.

Production backpressure controls

1. Bound admission: Set limits on outstanding compaction work by tenant, workflow, and queue. Admit a new task only when a capacity slot or explicit quota is available.
2. Coalesce equivalent requests: Deduplicate by tenant, workflow, source revision or event watermark, and agent role. A new revision may supersede a queued obsolete candidate, but must not erase the underlying committed events.
3. Separate urgent and background work: Give deadline-sensitive pre-handoff compactions priority over optional maintenance. Keep security review and publication capacity isolated.
4. Use queue-age deadlines: A compaction no longer useful after the next handoff should not run indefinitely. Mark it expired and route the workflow to an approved fallback, rather than processing a stale summary.
5. **Enforce *global* API quotas:** Coordinate request and token limits across Celery workers. A per-worker Celery task rate limit is not automatically a global distributed limit.
6. Limit retries: Retry transient 429 or connection failures with bounded exponential backoff, jitter, and provider-supplied retry guidance where available. Do not instantly requeue every failure.
7. Keep prefetch controlled: Long-running Celery tasks reserved far ahead can conceal backlog and starve urgent work. Tune worker_prefetch_multiplier and routing for task duration and broker behavior.
8. Use per-tenant fairness: Prevent one tenant's large ingestion burst from consuming all compaction capacity.
9. Protect downstream nodes: Before a model call, verify the context still fits and that its source facts, approvals, and version are valid.
10. Shed optional work first: Skip nonessential compaction or use targeted retrieval. Never silently skip authorization checks or treat a pending approval as passed.

What should an agent do when compaction is delayed?

A useful fallback hierarchy is:

• Reuse a validated summary plus a complete authorized event tail when it fits the model's safe context budget.
• Retrieve only the required source sections and build a role-specific context where that covers the task safely.
• Use a smaller, prevalidated working view if it retains all mandatory constraints.
• Delay or block the affected model call if no safe context can be assembled.

Do not use a partial, timed-out, or unvalidated summary just to keep the queue moving. The underlying workflow can remain resumable through persisted state.

Example: preventing a retry storm

Imagine 40 Celery compaction tasks simultaneously encounter an upstream quota limit. Without coordinated throttling, each immediately retries, creating another 40 requests, and potentially more timeouts.

Use a shared limiter or quota scheduler to admit only calls supported by remaining provider capacity, and return deferred tasks to a *bounded delayed-work mechanism*. Where possible, honor Retry-After, add jitter, and cap attempts. Investigate sustained errors instead of retrying indefinitely.

Do not assume ordinary Celery task_rate_limit coordinates a quota across every worker: documented task rate limits apply per worker instance. A broker-visible queue can also undercount work already reserved through prefetch.

LangGraph and Celery implementation

In LangGraph, trigger compaction at stable node or checkpoint boundaries, but keep the full authorized source evidence and durable state separate from the model-facing summary. A node needing a compact context should check whether a validated summary at the required revision is available and then select a safe fallback or pause. Checkpoint persistence does not itself enforce backpressure.

With Celery, isolate compaction queues, bound worker concurrency and prefetch, track queue delay separately from execution time, and put admission decisions ahead of task creation when possible. Use stable task IDs or logical deduplication keys. Compaction results must be idempotent and activated through version-aware comparison so late workers cannot overwrite newer summaries.

For example, an admission controller can follow this simplified logic:

PYTHON
def request_compaction(job):
    if validated_summary_satisfies(job):
        return "USE_EXISTING"
    if equivalent_job_pending(job):
        return "JOIN_PENDING"
    if tenant_quota_exceeded(job) or queue_at_capacity():
        return "DEFER_OR_SAFE_FALLBACK"
    enqueue_compaction(job)  # bounded, revision-keyed task
    return "QUEUED"

Actual implementation needs atomic admission accounting, expiry handling, and recovery of abandoned reservations. A check-then-enqueue sequence without concurrency protection can still over-admit bursts.

Trade-offs and metrics

• Hard queue limits protect capacity but can delay useful compactions; attach explicit workflow-safe fallback handling.
• Coalescing saves work but requires careful treatment of source versions and different role-specific contexts.
• Priority queues help critical handoffs but need aging or fair-share controls to avoid starving background tasks.
• Throttling reduces errors but increases queue wait, so context deadlines must remain visible.
• Additional workers help only when the true bottleneck is worker capacity, not the shared LLM quota.

Track admitted vs. deferred jobs, unique arrival/completion rates, oldest queue age, p95 wait, 429s, retry amplification, stale candidates, blocked workflow steps, safe fallback success, context overflow, and accepted-task quality.

How does this differ from earlier questions?

• Q181: General backpressure for communication and task flow across agents.
• Q237: Scheduling and prioritizing compaction work efficiently.
• Q238: Adding or removing compaction-worker capacity.
• This lesson: Controlling incoming demand and overload behavior when capacity cannot keep up, including admission limits, coalescing, retry suppression, and safe context fallback.

In short

When compaction demand exceeds sustainable capacity, regulate arrivals rather than just adding workers. Bound queues, deduplicate superseded work, honor global model quotas, limit retries, and preserve a verified path for agents to continue—or safely pause—without losing critical context.

Further reading

• Celery: Optimizing
• Celery: Task rate limits and retries
• LangGraph: Add and manage memory