AI Agents
An AI agent is a system that uses a model to choose actions in an environment. Agents often combine reasoning, memory, tools, retrieval, and feedback so they can complete multi-step tasks.
How do you coordinate memory compaction across concurrently running AI agents?
Coordinating memory compaction across concurrent AI agents means ensuring that a compacted summary represents a consistent set of agent updates, while no running agent loses work or sees partially published memory. Multiple agents may read the same working context, append observations, and complete branches in different orders. A compactor must not overwrite a newer result with an older summary. A useful design is immutable source events + versioned candidate summaries + validation + atomic activation . Beginner friendly example: API test generation agents Four agents generate test suites from an API specification: A requirements agent adds endpoint constraints. A test generator adds completed test case IDs. A security reviewer adds authorization findings. A compaction worker condenses accumulated history before the next model call. Suppose the compactor reads shared state at revision 12 . While it summarizes, the reviewer records a critical failed authorization test, creating revision 13 . If the compactor blindly saves its revision 12 summary as the latest context, the failed test disappears from the working memory. Read consistent revision → Generate candidate summary → Validate all source updates → Atomically activate only if revision still matches → Retry or reconcile on conflict Step by step numerical example Imagine three agents write concurrently: 1. At revision 40 , shared history contains 18,000 tokens . 2. The compactor begins creating a 5,000 token summary from revision 40. 3. The requirements agent adds 3 new constraints, and the reviewer adds 2 unresolved issues. The canonical revision advances to 42 . 4. Compaction based on revision 40 is now stale. Publishing it without protection would omit 5 important updates . 5. The compactor performs a conditional update: activate summary where current revision = 40. 6. Because the current revision is 42 , the update affects 0 rows ; the stale summary is rejected. 7. The compactor reads the five new updates, regenerates or safely extends the candidate, validates all 3 constraints and 2 issues , and attempts activation against revision 42. 8. If the revision still equals 42 and validation succeeds, the candidate becomes active. Otherwise, the worker repeats the conflict handling procedure. The exact summary length and agent count are illustrative; correct version checks matter more than the token savings. Three useful coordination strategies 1. Optimistic concurrency (usually the first choice): Let agents work normally, build a summary for a specified source revision, then use a database transaction or compare and swap to activate it. On conflict, retry from the latest source. This avoids stopping every agent but can waste compaction work during heavy updates. 2. Short coordinated pause: Temporarily stop writes for a particular workflow at a safe handoff or checkpoint, collect an agreed snapshot, compact, validate, then resume. It is simpler to reason about but introduces coordination latency and must not freeze unrelated tenants or tasks. 3. Event log/watermark compaction: Summarize an immutable sequence of updates through a known event watermark, such as events 1–250 . New events 251 onward remain available as an unsummarized delta. Readers receive summary through 250 + events after 250 . This is useful when continuous writes make a fully caught up snapshot difficult. A system can combine these strategies: compact a stable watermark, then atomically update a summary pointer after validating coverage. How do you avoid lost or duplicated updates? Use stable event IDs: Deduplicate retried or redelivered updates before they enter the source log. Record per event authorship: Store agent ID, workflow ID, tenant, source version, and event timestamp or sequence. Define merge rules: Independent sets of generated test IDs can be unioned with deduplication; an approval status cannot be merged with last write wins if it would overwrite an explicit rejection. Track exact coverage: Store the compacted source revision or event watermark range. Each event must appear either in the validated summary or the uncompacted tail, not disappear between them. Treat read after write explicitly: An agent that just committed a critical update should see at least that revision, or read authoritative state rather than a lagging summary. Separate candidate creation from activation: A completed summarization call is only a candidate. A transactional pointer change makes it the active version. Keep an immutable recovery path: Preserve raw facts and audit events according to retention rules. Do not delete them merely because the short context was published. A safe read model Store: active summary id covered through revision canonical revision pending events validation status When a reader assembles model context, load one consistent active summary version and append events after its covered revision through a captured read watermark. Check for gaps and duplicate event IDs. If the summary fails validation or critical events are missing, use the authoritative records or block the affected action. A summary pointer swap can be atomic in one database transaction. Updating a summary and a separate vector index in two independent systems is not automatically atomic; use revision aware indexing and make readers reject stale index entries. Example: compare and swap activation Illustrative SQL logic: Activate only if the candidate has already passed validation and exactly one intended row changes. The summary document and its validation record should be durable before the pointer moves. If the update affects zero rows, discard or recompute the stale candidate rather than forcing the write. In an event watermark design, a summary need not cover every newest event at activation; however, the reader must include the uncompacted tail and enforce any freshness requirements. What about parallel branches and checkpoints? If the test generator and security reviewer finish in parallel, wait for the intended branch join semantics or summarize through a durable event watermark that clearly identifies which branch updates are included. Preserve branch specific pending work and failed approvals. A compaction triggered in the middle of a tool call should not detach a tool response from its corresponding request or mistake an uncommitted effect for a completed one. For failed or retried work, distinguish the record of an attempt from a committed outcome . A retry must not duplicate published test suites or silently change approval state. LangGraph and Celery implementation In LangGraph , checkpointers persist thread level graph state, while a long term store can hold shared memory across threads. Use an application level revision or event log to coordinate memory compaction across separate threads; a graph checkpoint alone is not a distributed lock for all agents. Preserve compatible reducers and checkpoint boundaries when parallel branches merge. In Celery , put compaction into a retry safe task keyed by workflow, tenant, and source revision. Use a lease or unique work key to reduce duplicate compaction cost, but enforce correctness with transactional revision checks because leases can expire and tasks can be retried. A stale worker completion must never replace a newer active summary. Trade offs and monitoring Optimistic compaction minimizes pauses but may waste work under frequent writes. Paused compaction provides a stable snapshot but increases coordination latency. Watermark based compaction keeps writers active but requires reliable gap free event tracking. Monitor CAS conflicts, stale candidate rejection, compaction retries, event tail length, reader freshness lag, duplicate events, missing critical facts, and p95 context build latency . If too many candidates become stale, compact at quiet checkpoints or switch to a watermark based approach. How is this different from earlier questions? Q194: General consistency of shared state across agents. Q227: How to construct and publish compact working context. Q228: How to prevent loss of important information. Q229: How to evaluate compacted context quality. Q230: When to trigger compaction. This lesson: How concurrently executing agents coordinate which events enter a summary and when that summary becomes visible . In short Never let a compactor overwrite live agent updates. Snapshot a known revision or event watermark, retain new events, validate the candidate, activate it atomically, and assemble every reader's context from a consistent summary plus any uncovered updates. Further reading LangGraph: Add and manage memory LangGraph: Persistence and checkpoints
How do you recover from a failed memory-compaction operation in a multi-agent workflow?
Recovering from failed memory compaction means keeping the last trustworthy working context available, preventing partial summaries from becoming active, and rebuilding a valid compact context without losing agent updates. Compaction is an optimization—not an authoritative write of the task's underlying facts—so a compaction failure should normally leave durable workflow events, approvals, and artifacts intact. Beginner friendly production incident A four agent API test generation workflow uses a requirements agent, test generator, security reviewer, and publisher. After 30 minutes of work, a compactor starts shortening a long conversation for the next model call. The summarization model times out after generating only part of the summary. That partial output omits an unresolved OAuth scope issue. If the publisher reads the incomplete summary as authoritative, it might publish tests without the required authorization checks. Correct recovery: Reject the incomplete candidate, retain the last verified active context, inspect the source revision, retry safely when appropriate, and prevent the publisher from acting until its required facts and approvals are verified. Detect failure → Isolate candidate → Preserve canonical state → Classify cause → Retry or fallback → Revalidate → Atomically activate → Monitor Step by step numerical example Suppose the canonical event history contains 24,000 tokens and the last validated summary covers events 1–150 . Agents have since committed events 151–175 . 1. A compaction worker takes a snapshot through event 175 and attempts a new 6,000 token summary. 2. After 4 seconds , the summarization request times out. The incomplete candidate is not promoted. 3. While the worker is failing, agents commit 5 more events, advancing the canonical event watermark to 180 . 4. A retry using the old snapshot still covers only events 1–175 ; it cannot silently replace context that includes events 176–180 . 5. The safe immediate context is the last validated summary through 150 plus durable events 151–180 , assembled with revision aware reads. 6. The compactor generates a new candidate through 180 , checks required constraints and pending reviews, then publishes it with a conditional revision check. If the source advances again, retry or retain an uncompacted tail. 7. The system loses 0 committed events . The temporary token and latency cost can increase, but correctness is preserved. This example assumes durable, gap free event storage and consistent read semantics. A fallback that itself exceeds the model's token limit must use targeted retrieval, a smaller safe window, or block that model call rather than discard important facts. Distinguish common failure types Model timeout or rate limit: The summarizer never completes. Retry with bounded exponential backoff and jitter when transient, without changing the active summary. Malformed or incomplete output: JSON, schema, or required fields fail validation. Reject and regenerate with a stricter schema or smaller source partitions. Semantic validation failure: The candidate omits a critical constraint, changes approval status, or invents a source reference. Quarantine it for repair or review; do not blindly repeat an identical failing transformation indefinitely. Concurrent state conflict: Another agent writes new events before activation. Compare and swap rejects the stale candidate; reconstruct from the newest revision or a stable watermark. Database or storage failure: A candidate may have been written but its active pointer may not have changed. Read the committed pointer and validation status to determine whether the operation completed. Worker crash/redelivery: Celery may rerun a task after a crash. Use stable attempt IDs, candidate IDs, and idempotent writes to avoid duplicate promotion or corrupt state. Downstream regression discovered later: Revert the active summary pointer to the last validated version and append the uncompacted event tail, then investigate affected outputs and side effects. How do you make compaction crash safe? 1. Retain an immutable source of truth: Store original facts, event IDs, tool outcomes, source references, and approval decisions in durable storage under the proper retention policy. 2. Version every candidate: Record workflow ID, tenant ID, source watermark, compaction algorithm/prompt version, candidate ID, and creation attempt. 3. Use staged activation: Save candidate as PENDING, validate it, then mark VALIDATED; only a validated candidate may become the active summary. 4. Activate atomically: Use a transaction or revision checked compare and swap for the active summary pointer. A failed or stale attempt must not overwrite newer context. 5. Implement bounded retries: Define maximum attempts, backoff, timeouts, and a dead letter or review path for persistent errors. 6. Support safe fallback: Assemble context from a previous verified summary plus uncompacted events, or retrieve source evidence on demand. 7. Protect critical actions: Before publishing or writing externally, verify authorization facts and approval states against durable sources, not only compressed text. 8. Reconcile uncertain outcomes: After a timeout during database activation, read the authoritative pointer before retrying; do not assume a timed out operation definitely failed. 9. Keep tenant isolation: Every candidate, replay, and fallback must preserve tenant scope and retrieval permissions. 10. Test recovery paths: Inject model timeouts, crashes before/after pointer updates, stale revisions, corrupt summaries, and missing source records. Example of an atomic activation rule The compactor produces candidate s180 based on source revision 180 . An illustrative conditional update is: This is only one possible design. The candidate must already be durably stored and validated, and an application level transaction must preserve its relationship to the pointer. Zero rows updated means the expected revision did not match or the target is unavailable; do not force the promotion. For a watermark plus tail design, readers may safely use a validated summary through an earlier revision if they also include every authorized event after that watermark through a consistent read boundary. When should retries stop? Retry transient errors, such as connection timeouts, with limits. Stop automatic promotion on safety or semantic failures. For example, if three attempts omit the same mandatory approval, escalating for review is safer than accepting the shortest output. A fallback should be observable and bounded: continuing with unlimited raw history could repeatedly hit the context limit or increase cost unexpectedly. LangGraph and Celery example In LangGraph , keep checkpoints and durable facts separate from the model facing summary. Save validated summaries in versioned state or a store, and use a graph node to choose the active context plus unsummarized tail. Test restoring a checkpoint after a compaction crash; a checkpoint is not a guarantee that an externally stored summary is still valid. With Celery , give each compaction job a stable key such as tenant id + workflow id + source revision. Mark results idempotently, use retry/backoff for transient failures, and always check the source revision before activation. A lease can reduce duplicate compaction work, but cannot replace transactional correctness if it expires or a worker is redelivered. Metrics and trade offs Track compaction failure rate, retries per candidate, stale candidate rejections, recovery latency, fallback context length, context overflow errors, missing event incidents, and downstream task regressions . Keeping previous summaries and raw events uses more storage, and safe fallback can temporarily increase tokens and latency. Those costs are usually preferable to lost decisions or unsafe publications. Avoid retaining sensitive raw content longer than the approved retention policy. Difference from earlier lessons Q227: Implements compaction. Q228: Defines information preservation checks. Q229: Evaluates quality. Q230: Chooses trigger timing. Q231: Coordinates concurrent updates and atomic activation. This lesson: Recovers from timeouts, invalid candidates, worker crashes, ambiguous commits, and post promotion defects without dropping committed agent work. In short Treat compaction output as a replaceable, validated cache of durable evidence. On failure, leave the active context unchanged, keep all newer events, retry only when safe, and promote a new summary through a version checked atomic operation.
How do you version and audit compacted memory across long-running multi-agent workflows?
Versioning and auditing compacted memory means recording exactly which source events, agent decisions, and validation results produced each shortened working context—and which version every agent actually consumed. This makes long running multi agent workflows reproducible, debuggable, and recoverable when an important detail disappears or a later output is disputed. A compacted summary is a derived view , not the authoritative record. Keep the source events and artifacts available according to approved retention policies, and store an immutable audit record for each candidate and activation. Beginner friendly example: API test generation A LangGraph workflow uses a requirements agent, test generator, security reviewer, and publisher. Each agent contributes facts about POST /payments, including OAuth scopes, generated test IDs, unresolved bugs, and approval outcomes. After 80 messages, a compactor creates a short summary. Three days later, someone asks why the publisher omitted a 401 Unauthorized test. Without version history, the team sees only the latest summary and cannot tell whether the original requirement was missing, the compactor removed it, or the downstream generator ignored it. With an audit trail, the team can reconstruct the source revision → compacted summary → validation → active version → consuming agent → published artifact . Record source lineage → Build immutable summary candidate → Validate → Atomically activate → Log which version agents read → Reproduce or roll back when required Numerical example Suppose a workflow has 240 committed events . 1. Summary S1 includes events 1–100 , created using compactor version c1 . 2. Summary S2 includes events 1–180 , created using version c2 after 80 additional events were committed. 3. Summary S3 includes events 1–220 , created using version c2 and a revised validation rubric. 4. The active working context is S3 + events 221–240 . It must not silently lose the 20 event uncompacted tail . 5. The publisher's audit entry shows it read S3 , saw the tail through event 240 , and produced artifact T57 . 6. If the security reviewer later discovers an omitted scope, the audit trace identifies the summary and exact source event IDs needed to compare the original requirement against the context consumed by the publisher. A summary's coverage watermark is not the same as the canonical event revision. A summary through event 220 is still useful when event 240 exists, but readers must include relevant later events or use a freshness guard. What metadata should each compacted version contain? 1. Identity: Tenant ID, workflow/thread ID, immutable summary ID, creation time, and producing compaction job ID. 2. Source lineage: Canonical source revision or event range, content/artifact references, and any excluded categories of ephemeral data. 3. Compactor configuration: Model identifier, prompt/template version, algorithm version, token budget, and relevant tool or schema version. 4. Validation record: Validator version, mandatory field coverage, groundedness checks, open issue preservation, and pass/fail reasons. 5. Content integrity: Hash of a canonical serialized summary plus a record of where the summary is stored. 6. Lifecycle status: CANDIDATE, VALIDATED, ACTIVE, SUPERSEDED, REJECTED, or QUARANTINED, as appropriate. 7. Activation details: Actor/worker, exact time, prior active version, accepted source revision, and successful conditional write result. 8. Consumer trace: Which agent run and model call used which summary ID, event tail watermark, and source references. Keep secrets and personal data out of ordinary audit logs; use scoped references and cryptographic hashes with suitable access controls. Simple example of a versioned summary record The digest is illustrative. In production, generate it from the actual canonical bytes; never write a placeholder hash into a real integrity check. Candidate version versus active version Treat these separately: Candidate: A newly generated summary that may be incomplete or unsupported. Validated version: A candidate that passes integrity, completeness, and safety checks. Active version: The validated summary selected by an atomic pointer update for authorized readers. A database transaction or compare and swap should activate the correct summary while checking the expected current revision. A worker completing late must not replace a newer version. Keep activation and rejection history append only so later incident analysis can see what happened. If an event watermark design is used, allow a validated summary covering an older watermark while ensuring readers append the missing event tail. If strict read after write consistency is needed, require a reader's consumed revision to be at least the revision it just committed. How do you audit an unexpected output? Imagine the published test suite omitted an authorization test. 1. Find the published artifact ID and its agent execution ID. 2. Read the consumer trace to identify the exact summary version and event tail watermark. 3. Compare the original API specification with the committed requirements events. 4. Check whether those facts existed in the compacted summary or the consumed tail. 5. Inspect the compactor's validation results and any waived checks. 6. Distinguish source extraction failure , compaction omission , retrieval/assembly failure , and downstream reasoning failure . 7. Re run the workflow in a controlled environment using the preserved inputs, compatible model/tool versions, and the recorded prompt configuration. Reproduction is best effort: Model services, nondeterministic generation, and changing external APIs may prevent bit for bit identical outputs, even when the lineage is complete. Store snapshots of external tool results where policy permits. Retention and privacy trade offs Auditability does not mean retaining every raw message forever. Apply approved retention periods to full transcripts, tool inputs, and summaries; store minimal provenance for longer where lawful and useful. Honor deletion obligations across source and derived records, and explain when a historical replay is no longer possible because source content was legitimately deleted. When a memory is purged, update indexes and references so agents do not keep retrieving it from an old compacted summary. A hash can support integrity but does not prove that the summary is factually correct. LangGraph and Celery implementation In LangGraph , record graph release version, thread/checkpoint identifier, and summary ID together in the execution trace. A checkpoint helps restore state, but the model facing context may be a selected summary plus recent messages; log exactly what was supplied. If a thread resumes after an upgrade, ensure the summary schema and source references remain compatible. In Celery , use durable, idempotent compaction jobs keyed by tenant/workflow/source revision. Log task attempt IDs and record validation and activation outcomes separately. Redelivered or older tasks may finish after a newer summary is active: conditionally reject stale activation instead of rewriting history. Useful operational metrics Track summary versions per workflow, active summary age, unconsumed event tail length, stale activation attempts, validation failures, unexplained source gaps, audit trace completeness, and time to reconstruct a faulty output . Establish explicit alerts for missing provenance or consumers using an unvalidated summary. Difference from previous lessons Q227: Builds a compact working context. Q228: Preserves critical information during compaction. Q229: Measures compaction quality. Q230: Determines when compaction is triggered. Q231: Coordinates concurrent updates. Q232: Recovers from compaction failures. This lesson: Establishes immutable version history and end to end lineage so a team can explain, compare, and audit what agents actually saw. In short Version every compacted summary, connect it to exact source events and validation results, and record which version each agent used. Use atomic activation and permission aware retention so compacted memory remains both operationally efficient and auditable.
How do you manage retention and deletion of compacted memory versions in multi-agent systems?
Managing retention and deletion of compacted memory versions means deciding which historical summaries must remain available for active workflows, audits, or recovery, and which can be removed without breaking agents or exposing deleted information. Unlike ordinary memory TTL, compacted summaries can form a version history : one version may still be referenced by a running workflow, an evaluation result, or a rollback pointer. The safest approach is version aware retention, reference tracking, tombstones, and verified cleanup . Never delete a version solely because a newer summary exists. Beginner friendly example: API test generation agents A requirements agent, test generator, security reviewer, and publisher share a compacted history for POST /payments. Summary S1 includes events 1–100. Summary S2 includes events 1–180. Summary S3 includes events 1–240. A paused security review workflow still references S2 , while new workflows use S3 . Deleting S2 immediately could break the paused review or make its audit trace unreproducible. But retaining every summary forever increases storage cost and can retain personal or sensitive information longer than authorized. Inventory versions and references → Apply retention and deletion rules → Block obsolete reads → Rebase active workflows → Purge eligible versions → Verify derived copies Numerical example Assume a project contains 1,000 historical summary versions : 1. 100 are current active versions. 2. 150 are superseded but still referenced by paused workflows or approved audit retention. 3. 50 are under an applicable preservation hold. 4. 700 are superseded, unreferenced, and past the approved retention period. If the groups are mutually exclusive, the cleaner may mark 700 / 1,000 = 70% as deletion candidates. If each occupies 40 KB , eligible stored payload is 700 × 40 KB = 28,000 KB , approximately 28 MB in decimal units. This does not mean 28 MB is immediately reclaimed: databases, snapshots, backups, indexes, and object store retention have separate cleanup schedules. First confirm that no protected reference or policy prevents deletion. Retention rules for versioned summaries 1. Identify the version: Track tenant, workflow/thread ID, summary ID, source event watermark, creation time, and content hash. 2. Record references: Maintain the active summary pointer, paused checkpoint references, audit dependencies, rollback dependencies, and any explicit retention holds. 3. Define policies by purpose: Working summaries may have short operational retention; audit records may require longer retention. Apply current business, privacy, contractual, and legal requirements rather than assuming a universal period. 4. Separate content from provenance: Where permitted, preserve minimal non sensitive lineage metadata without retaining full raw text indefinitely. 5. Use deletion states: Mark a version ACTIVE, SUPERSEDED, DELETE PENDING, or DELETED with immutable audit events and a verified completion record. 6. Require safe dereferencing: Move or revalidate paused workflows before deleting their referenced versions; do not silently switch them to a newer summary with different meaning. 7. Handle sensitive deletion requests: Invalidate relevant summaries immediately in authorized reads, trace derived copies containing the information, and honor applicable deletion obligations and exceptions. 8. Clean derivatives: Remove or invalidate vector entries, cached snippets, materialized views, stored evaluations, and authorized exports where required and feasible. 9. Make cleanup idempotent: Stable summary IDs and generation/version checks stop retried tasks from accidentally deleting an unrelated replacement. 10. Verify and monitor: Check remaining live references, unexpected retrieval hits, cleanup failures, and backup lifecycle limitations. Why ordinary reference counting may be insufficient Suppose S2 has no direct active summary pointer, but a paused LangGraph checkpoint still contains summary id=S2. A cleaner that counts only currently active readers could delete S2 prematurely. More subtle: an evaluation artifact might depend on S2 as its source, or an older summary might be needed for a documented rollback. Before removal, account for these durable references and any retention rule. A lease can protect short lived readers, but it is not a substitute for a persistent reference from a paused workflow. Reference count zero is a useful signal, not sufficient permission to delete. Expiration, holds, privacy rules, and downstream dependencies still apply. What if a person requests deletion of information that appears in many summaries? Deleting only the original memory record is insufficient if S1, S2, S3, and search indexes still contain the same sensitive text. A safe process is: Identify source IDs and summary lineage that incorporated the data. Revoke or filter affected summaries from future retrieval while cleanup proceeds. Rebuild required current summaries from permitted remaining sources, or block affected workflows until safe. Purge eligible derived content and associated caches/index entries. Record minimal deletion evidence without copying the deleted content into the audit log. Account for backup retention and applicable legal preservation requirements. Cryptographic hashes and references should be reviewed for privacy impact; a hash alone does not guarantee anonymity. Safe cleanup under concurrency At 10:00, a cleaner reads S2 as unreferenced. At 10:01, a workflow tries to pin S2. If these actions are not coordinated, the cleaner might delete a version that the workflow has just selected. Use a transactional state transition: first mark S2 DELETE PENDING with a generation check and prevent new references; then confirm existing references are absent before physical deletion. The workflow must atomically pin only a version still eligible for reading. If that fails, it must choose an authorized alternative and revalidate its required context. Across a database and an external vector store, physical deletion is generally not one atomic transaction . Use durable cleanup jobs, tombstones, and read time filters so a stale index cannot make deleted content available again. LangGraph and Celery example With LangGraph , record which compacted summary version each checkpoint consumed. A resumable thread may need that version even after it is superseded. Before deletion, verify that the thread has completed, been validly migrated, or no longer requires the content; comply with any overriding deletion policy. With Celery , run deletion in idempotent bounded batches. Include tenant ID, summary ID, and version/generation in each task. Retry failed storage or index cleanup, and never allow a delayed cleanup task to remove a newly re created version with a different generation. What do you measure? Useful metrics include number of superseded versions, protected references, eligible deletion candidates, oldest retained version age, deletion completion lag, orphaned cache/vector entries, blocked reads of tombstoned summaries, and failed cleanup retries . Audit the decision to retain or remove a version, not only the physical deletion event. How does this differ from previous lessons? Q226 (expiration and garbage collection): When ordinary memory records expire and how their copies are cleaned up. Q233 (versioning and auditing compacted memory): How summary history and provenance are recorded. This lesson: How to safely retire particular historical summary versions while respecting paused workflow references, audit dependencies, deletion requests, and cross store cleanup. In short Retain only the compacted versions still needed or legally required, block new references before deleting them, and verify cleanup across every derived store. Historical summary deletion is a dependency and privacy problem—not simply deleting the oldest row.
How do you handle memory compaction when multiple agents have different context requirements?
When multiple AI agents need different information, compact shared memory into a verified common core plus role specific views instead of forcing every agent to consume one generic summary. A requirements extractor needs precise source rules, a test generator needs testable cases, and a security reviewer needs authorization controls and unresolved findings. The shared core preserves facts and handoff state; each agent's view emphasizes the evidence it needs. A role specific view is a derived projection , not a separate source of truth. All views should reference the same authoritative records and versions. Beginner friendly example: API test generation A workflow uses four agents: Requirements agent: Needs endpoint paths, schemas, response codes, source sections, and open ambiguities. Test generation agent: Needs acceptance criteria, valid/invalid input boundaries, existing test IDs, and coverage gaps. Security reviewer: Needs authentication scopes, permissions, tenant isolation rules, and unresolved vulnerabilities. Publisher: Needs verified artifacts, final approval status, destination, and idempotency key. A generic 4,000 token summary might mention “OAuth2 required” but omit the exact pay:write scope. That may be acceptable background for some tasks, but not for security review or publication . Persist canonical evidence → Define each agent's required fields → Create a shared core → Build role specific projections → Validate and authorize each view → Assemble context at handoff Step by step numerical example Suppose the uncompacted shared history is 24,000 tokens . The pipeline makes one call to each of four agents, and each initially receives all 24,000 tokens. 1. Replayed context across the four calls totals 4 × 24,000 = 96,000 input tokens . 2. A validated 3,000 token common core retains workflow identity, source versions, decisions, and pending work. 3. The requirements view adds 5,000 tokens , the test generator view 4,000 , the security view 6,000 , and the publisher view 2,000 . 4. The four per agent contexts become 8,000, 7,000, 9,000, and 5,000 tokens . 5. Total replayed context is 29,000 tokens , a reduction of 67,000 / 96,000 ≈ 69.8% . 6. This is not automatically the net cost saving. Add the tokens spent creating, validating, refreshing, and retrieving projections, and account for input/output pricing and caching. 7. If the security projection omits even one mandatory OAuth scope , reject that view despite the apparent 69.8% reduction. These numbers are illustrative and assume the role specific views contain all relevant information for the modeled calls. Design a two layer memory structure Layer 1: authoritative shared core Store workflow and tenant IDs, source document versions, artifact IDs, committed decisions, critical global constraints, pending approvals, and the event watermark. This layer should be consistent across agents and should never depend on a free form summary alone for authorization. Layer 2: role specific compact projections Generate versioned views keyed by workflow id + agent role + source revision + projection version. For example, a security view references the source IDs for OAuth requirements, while the publisher view references approval and artifact IDs. Both can retrieve exact underlying evidence when needed. A role specific view should not invent a new “fact” that conflicts with the canonical core. Practical implementation 1. Specify each agent's contract: Define mandatory fields, optional context, forbidden fields, freshness requirements, and output expectations. 2. Separate global from local facts: Keep hard constraints and cross agent state in the shared core; include specialist details only where relevant. 3. Use structured projections: Prefer source IDs, requirement IDs, approvals, and status objects alongside concise natural language explanations. 4. Validate role specific completeness: Check requirements against each role's rubric. The test generator may require boundary conditions; the publisher must require verified approvals. 5. Enforce access controls before summarization: A projection must not aggregate records the consuming agent or tenant is not allowed to read. Redact restricted fields rather than relying only on prompting. 6. Track provenance and revisions: Each view records source watermark, compactor configuration, and validation result. 7. Assemble per call context: Supply the common core, the current role view, selected recent messages, and on demand source excerpts—not every view to every agent. 8. Invalidate on relevant changes: When a security requirement changes, refresh the security projection and any downstream view that depends on it. 9. Support safe fallback: If a needed fact is unavailable or stale, retrieve approved authoritative evidence or block the affected action rather than guessing. 10. Observe cost and quality: Measure projection size, refresh lag, missed requirements, extra retrievals, and task success per role. Example of a safe security projection The security agent can use this compact structure while retrieving the exact specification section if a decision requires more detail. The publisher must independently confirm the final approval status from the authoritative source; it should not treat an old projection as approval. How do you handle different context windows and budgets? Agents may use models with different usable context capacities. Assign per role budgets after reserving space for system instructions, tools, retrieved evidence, and output. If the security agent needs more evidence than its budget allows, prioritize required source excerpts and drop low value conversational repetition. Never resolve a tight budget by silently removing mandatory constraints. A role can also use a lightweight view for planning and a richer evidence view for a critical decision. The model facing view is not the full durable memory. Concurrent updates and version safety Imagine the requirements agent commits a new permission rule at source revision 19 while the security projection still points to revision 18 . The security reviewer must check the required source revision or retrieve the new authoritative rule. A compaction worker creating a projection from revision 18 must not overwrite a validated revision 19 projection. Use compare and swap or immutable versioned views and atomic pointers. For event log designs, a view may cover an earlier watermark if the consuming agent also receives a complete, authorized event tail through the required read boundary. Different context views must not become inconsistent copies of truth. LangGraph and Celery integration In LangGraph , keep durable workflow state and shared memory records separate from the prompt assembled for each node. Each agent node can read the canonical core and its authorized role specific projection, then retrieve exact evidence as needed. Store projection IDs and source revisions in traces for reproducibility. With Celery , build projections as idempotent jobs keyed by tenant, workflow, role, and source revision. Coalesce duplicate rebuild requests, but validate revisions before activating results; an old retry must not replace a newer view. Ensure queue messages carry compatible role and version metadata. Trade offs and common mistakes Single summary for everyone: Simple, but it may hide details required by specialist agents. Independent summaries with no shared core: Flexible, but contradictory facts and approvals can appear across roles. Excessive specialization: Many projections increase storage, refresh cost, and invalidation complexity. Over sharing: Sending every specialist view to all agents increases tokens and may violate least privilege. Trusting a stale projection: A compact view can be well formed yet outdated; check authoritative revisions before critical actions. A good baseline is one verified core and only the specialized projections that show measurable quality or efficiency benefits. Difference from earlier lessons Q165: Explains context sharing between agents generally. Q227–230: Cover compaction implementation, preservation, evaluation, and triggers. Q231–234: Cover concurrent updates, failure recovery, audit/version history, and deletion. This lesson: Designs different authorized compact contexts for different agent roles , while retaining one authoritative shared core. In short Give every agent the context it needs—not all the context available. Maintain a verified common core, tailor compact projections by role, preserve exact evidence references, and enforce revision and permission checks before each critical handoff.
How do you measure the cost and latency impact of memory compaction across multiple AI agents?
Measure memory compaction performance at the level of the entire multi agent workflow, not merely by counting how many prompt tokens disappear. Compaction can reduce repeated input tokens but introduce summarization calls, validation, cache misses, retrievals, retries, and queue delays. The question is whether the completed, correct task becomes cheaper or faster. In practice, compare the same workflow with compaction disabled and compaction enabled using consistent model versions, workload mix, cache policy, and quality checks. Beginner friendly production example A LangGraph pipeline generates API test suites using a requirements agent, test generator, security reviewer, and publisher. The history grows as each agent adds tool results and findings. Without compaction, every subsequent model call rereads much of that history. With compaction, the coordinator builds a validated common summary and small role specific context. A team notices that input token usage falls sharply, but the API endpoint's p95 completion time rises because compaction jobs queue behind other Celery tasks. Looking only at prompt token reduction would hide the regression. Define workflow baseline → Instrument every stage → Run comparable workloads → Calculate fully loaded cost → Compare latency distributions → Check quality → Decide whether to retain compaction Step by step numerical example Assume one completed test generation workflow makes 8 downstream LLM calls . The figures below are illustrative, not current provider prices. Without compaction Each downstream call receives 20,000 history tokens : 8 × 20,000 = 160,000 input tokens. It generates 12,000 output tokens total . At an illustrative input price of $2 per million tokens and output price of $8 per million , model cost is 160,000/1,000,000 × $2 + 12,000/1,000,000 × $8 = $0.416. End to end duration is 80 seconds . With compaction The downstream calls receive 6,000 history tokens each , or 48,000 tokens total. The compactor itself consumes 22,000 input tokens and produces 2,000 output tokens . Validation and fallback retrieval require another 10,000 input and 1,000 output tokens . The downstream agents still generate 12,000 output tokens . Fully loaded input is 48,000 + 22,000 + 10,000 = 80,000 tokens; output is 12,000 + 2,000 + 1,000 = 15,000 tokens. Model cost is 80,000/1,000,000 × $2 + 15,000/1,000,000 × $8 = $0.28. Model cost savings = ($0.416 − $0.28)/$0.416 ≈ 32.7% . End to end duration is 87 seconds because compaction, validation, and queueing add more wall clock time than downstream calls save. Conclusion: In this example, model spending drops by 32.7% , but latency worsens by 7 seconds (8.75%) . If the user facing latency SLO is strict, compaction is not an unconditional win. The calculation also excludes storage, CPU, telemetry, and infrastructure expenses; measure those for total cost of ownership. Which metrics should you capture? 1. End to end cost per accepted workflow: Total model charges plus relevant compaction infrastructure, retrieval, vector store, and retry expenses; divide by accepted completed tasks, not by attempts. 2. Model token usage: Input and output tokens per agent, compactor, validator, and fallback request; separate billable cached input tokens if the provider exposes them. 3. Compaction overhead: Candidate generation tokens, duration, validator calls, retries, rejected summaries, and queue wait time. 4. Context savings: (baseline repeated context tokens − new repeated context tokens) / baseline repeated context tokens. Label this a gross context reduction , not a net billing saving. 5. Latency distribution: End to end p50, p95, p99, time to first useful response when applicable, and per node service/queue durations. 6. Throughput and resource load: Accepted workflows per minute, concurrent executions, Celery backlog, and pressure on model and storage rate limits. 7. Fallback and quality: Extra retrieval requests, missing requirement rate, failed security checks, human rework, and successful completion rate. 8. Cache behavior: Prompt cache hit and billed token changes, because shortening prompts can sometimes reduce the value of existing caching. How do you measure latency correctly? Record the wall clock times of workflow start, compaction queueing, compaction execution, validation, each downstream model/tool step, and final acceptance. For parallel branches, do not add all node durations to estimate end to end latency ; overlapping nodes make summed durations larger than the critical path wall time. Example: requirements and security analysis run in parallel for 12 and 18 seconds. Together they contribute approximately 18 seconds , not 30 seconds , to the critical path if there is no additional serial work. A compactor blocking both branches can lengthen the critical path, whereas compaction running asynchronously ahead of a later handoff may not. Watch p95 rather than only average latency: retries, queue spikes, and very long contexts often produce tail latency that averages conceal. Set up a fair comparison Paired offline benchmark: Run the same set of API endpoints with full versus compacted contexts. Hold model versions, inputs, tool permissions, and acceptance rules constant. Repeat stochastic cases where feasible. Shadow evaluation: Generate compacted contexts without changing production decisions, then evaluate their correctness and expected token cost. Canary or randomized rollout: Compare production cohorts with similar endpoint complexity, workload, time of day, caching, and concurrency. Keep a fast rollback route. Stratify results: Measure short versus long conversations, high versus low agent count, security heavy workflows, and low versus high context reuse. Include failures: Count compaction timeouts, failed validations, retries, and fallback paths. Excluding them biases the claimed savings upward. Do not benchmark only workflows that successfully compacted. Use the complete eligible cohort, including failed or skipped attempts. When does compaction pay for itself? A simple break even estimate is: net savings = (avoided downstream model charges) − (compaction + validation + fallback + infrastructure charges). For example, if generating and checking a summary costs $0.04 and each future agent call saves $0.012 in model charges, then after 4 future uses , gross avoided charges are $0.048 , producing $0.008 net savings before other overhead. After only 2 uses , savings are $0.024 , so compaction loses $0.016 . This shows why a compaction policy should consider the expected number of future reuses, not just the length of the current transcript. LangGraph and Celery implementation In LangGraph , record a trace for the workflow and child spans for agent nodes, summarization, memory retrieval, validation, and model calls. Include workflow ID, summary version, source revision, and a compacted/not compacted flag. Be careful not to expose sensitive prompt content or personal data in telemetry. In Celery , separately time broker queue delay and worker execution time for compaction tasks. Collect retry counts, failure reasons, worker saturation, and time to publish a validated summary. Correlate task IDs back to the LangGraph run so queue time is not mistaken for model latency. OpenTelemetry's evolving generative AI conventions describe token usage and operation duration measurements; tools such as LangSmith can provide agent traces and cost/latency analysis. Instrumentation completeness and actual billed cost should be verified against the chosen model provider. Production decision rule Adopt compaction only if it satisfies all required constraints : No unacceptable regression in test suite correctness or critical security coverage. Lower fully loaded cost per accepted workflow for the intended workload. p95 end to end completion time remains within the application's SLO. Failure and fallback rates remain within agreed budgets. Audit and tenant isolation requirements stay intact. A system optimized for batch generation might accept seven extra seconds to save money. An interactive assistant with a strict response time target might choose fewer compactions or move them off the critical path. Difference from earlier questions Q202: General strategies for reducing multi agent workflow cost. Q203: General multi agent performance metrics. Q229: Whether a compacted summary is accurate and useful. Q230: When compaction should trigger. Q235: Tailoring compacted context by agent role. This lesson: Measures the actual marginal cost, critical path latency, queue impact, and break even point of memory compaction in production. In short Count the whole workflow. Saving prompt tokens is valuable only after accounting for compactor and validator charges, retrieval fallbacks, queueing, caching effects, quality regressions, and end to end latency. Further reading OpenTelemetry: Generative AI metrics LangSmith: Observability and evaluation LangGraph: Memory and summarization
How do you optimize memory-compaction scheduling to reduce bottlenecks in multi-agent systems?
Optimizing memory compaction scheduling means deciding when compaction jobs should run, how much worker capacity they may use, and which jobs deserve priority so summarization does not block the agents it is meant to help. The key is to keep compaction off the user visible critical path where possible while still completing it before a workflow would exceed its safe context budget. Unlike choosing a trigger threshold for a single workflow, scheduling looks across many workflows competing for workers, rate limits, and storage throughput. Beginner friendly example: multi agent API test generation A system uses LangGraph for requirements extraction, test generation, security review, and publication. Celery workers run 50 concurrent workflows. Each workflow may request a compaction after accumulating tool results. If all 50 compactions arrive at once and share a worker queue with security reviews, compaction can delay higher priority review tasks. Even though the resulting prompts are shorter, the entire test generation service may become slower. Forecast demand → Classify urgency → Choose queues and worker limits → Coalesce duplicate jobs → Compact ahead of handoffs → Monitor lag and quality → Adapt scheduling Numerical example: avoiding a compaction queue Suppose 60 workflows each request one compaction taking an average of 6 seconds of worker execution, and the service has 3 dedicated compaction workers . 1. Total work is 60 × 6 = 360 worker seconds . 2. With perfectly balanced, uninterrupted execution, the theoretical minimum processing time is 360 / 3 = 120 seconds . 3. If all requests arrive together and each worker processes 20 jobs sequentially, the last job finishes after approximately 120 seconds ; its wait before starting is roughly 114 seconds . 4. During that wait, some workflows may hit context budget limits. Adding more retries to the same queue would make congestion worse. 5. If 40 requests are duplicates or low value because the same source revision has already been compacted, coalescing them leaves 20 distinct jobs . 6. At the same assumed service time and concurrency, processing 20 jobs takes roughly ceil(20/3) × 6 = 42 seconds in balanced waves, without counting real world overhead. The savings come primarily from removing redundant work, not from assuming that queueing itself improves model speed. Actual service times vary; monitor p95 job duration and external model rate limits rather than treating these estimates as guarantees. What scheduling strategies work? 1. Separate compaction capacity from critical agents: Use a dedicated Celery queue or bounded concurrency so background summaries cannot consume every worker needed for validation or publication. 2. Classify urgency: Prioritize workflows projected to exceed their next step context budget. Compactions for workflows with ample room can be lower priority. 3. Precompute at safe boundaries: Compact after a stable checkpoint or before a known expensive handoff, ideally while another independent task can proceed. 4. Coalesce duplicate requests: Use a key such as tenant id + workflow id + source revision + projection role. One validated result can serve several equivalent requests when permissions and context requirements match. 5. Debounce bursts: Wait briefly for rapid updates to settle if the workflow has sufficient context headroom. Do not debounce past a hard context safety deadline. 6. Bound queue and rate limit pressure: Apply per tenant fairness, caps on concurrent model calls, and backoff with jitter for transient rate limits. Do not let one tenant monopolize the compactor. 7. Schedule by expected value: A summary reused by ten upcoming agents can be more valuable than one produced for a workflow one step from completion. 8. Use incremental compaction where safe: Summarize only newly committed events and merge them with a verified version, while periodically checking against authoritative sources to avoid accumulated summarization errors. 9. Handle overload explicitly: Fall back to targeted source retrieval, smaller authorized context, or a blocked step if no validated summary can be produced before the limit. 10. Keep source revisions in every job: A stale or late compaction task must not activate a summary that omits newer agent updates. Example: priority score For illustration, classify candidates using three signals: Context urgency: How close the next model call is to the usable token limit. Expected reuse: How many future agent calls benefit. Workflow criticality: Whether the next consumer is blocked without a safe summary. A task that must compact before the very next security review is higher priority than a nightly cache refresh. Use an explicit upper bound on waiting time and fair share rules so low priority work is not starved indefinitely. A numeric scoring formula is optional; clear priority classes plus aging often work well. Queueing versus critical path latency The compaction service's execution time and queue wait time are different measurements. A compaction may take 6 seconds to execute but wait 90 seconds in the queue. If the next agent cannot proceed without that summary, the workflow experiences nearly 96 seconds of compaction related delay. If the compaction completes while unrelated branches are running and is ready before the next handoff, most of that work may not add to end to end latency. For parallel LangGraph branches, measure the critical path , not the sum of all task durations. Safe schedule aware fallback Before each model call, check the projected token budget and the availability of a validated summary covering the required source revision. If a compaction job is queued but not complete, do not treat its pending candidate as usable. If the safe context fits, proceed with the last validated summary plus the complete new event tail. If it does not fit, use scoped retrieval or delay/block the affected call. Never silently omit mandatory authorization rules or unresolved approvals to maintain throughput. LangGraph and Celery implementation In LangGraph , create a compaction task at a stable node boundary and persist its source revision and intended consumer. A downstream node can check whether the validated summary exists; if not, it chooses a safe fallback path. Checkpoint persistence does not itself guarantee that the background summary has finished. In Celery , route compaction tasks to a distinct queue with explicit worker concurrency, retries, deadlines, and deduplication keys. For long tasks, understand task acknowledgment and redelivery behavior; make activation idempotent and conditional on the source revision. Avoid over reserving tasks if that prevents urgent jobs from reaching available workers. Capacity plans must account for shared LLM API rate limits even when local Celery concurrency is increased. What should you measure and tune? Compaction queue depth, queue age, p50/p95 wait and execution times. Number of coalesced/skipped jobs, retries, rate limit errors, and stale results. Percentage of workflows blocked waiting for compaction. Context overflow failures and safe fallback usage. End to end p95 completion time, cost per accepted workflow, and output quality. Per tenant fairness and starvation incidents. A rollout should compare these before and after changing scheduling policy. Faster compaction throughput is not a success if it degrades security review quality or starves higher priority work. Difference from previous lessons Q180–181 and Q198: General bottlenecks, backpressure, and scheduling fairness. Q230: When one workflow should request compaction. Q231: How concurrent compaction results remain consistent. Q236: How to measure compaction's cost and latency. This lesson: How to schedule compaction work across many workflows using capacity isolation, prioritization, coalescing, and overload controls. In short Run compaction early enough to protect future context budgets, but isolate and prioritize it so it never overwhelms critical agent work. Measure queue wait, critical path impact, fairness, and correctness—not merely how many summaries per minute the workers produce. Further reading Celery: Routing Tasks Celery: Optimizing LangGraph: Persistence
How do you autoscale memory-compaction workers in a multi-agent system?
Autoscaling memory compaction workers means adjusting the number of workers that build validated summaries as demand rises or falls, without starving other agents, exceeding model provider limits, or allowing stale summaries to overwrite newer state. Autoscaling handles capacity ; it does not replace sound scheduling, quality checks, or revision safe activation. Beginner friendly example A LangGraph workflow has a requirements agent, test generator, security reviewer, and publisher. Celery workers handle background compaction. During a quiet period, two workers keep up. After a large API spec upload, 120 workflows request summaries within minutes, and the queue grows. The platform can temporarily increase compaction worker replicas, then reduce them as the queue drains. But if all replicas call the same LLM provider, launching more workers may simply produce rate limit errors. Measure incoming demand → Estimate capacity → Check external limits → Scale bounded workers → Preserve safe retries → Evaluate backlog and workflow latency → Scale down gradually Numerical example: sizing workers Suppose: New compaction jobs arrive at 24 jobs per minute . One worker has an average job execution time of 10 seconds . The team aims to keep utilization at or below 75% to leave headroom for variability. One worker's theoretical service capacity is 60 / 10 = 6 jobs per minute . At 75% target utilization, its planned sustainable capacity is 6 × 0.75 = 4.5 jobs per minute . Required workers ≈ ceil(24 / 4.5) = 6 . With only 3 workers , planned sustainable throughput is 3 × 4.5 = 13.5 jobs/minute ; at the stated arrival rate, backlog would grow by approximately 10.5 jobs/minute , assuming the workload and service time remain steady. With 6 workers , planned capacity is 27 jobs/minute , leaving modest headroom. However, if an external LLM API permits only 18 compaction calls per minute for this workload, six workers cannot sustainably complete 24 calls/minute. The true bottleneck is the provider limit . The system must reduce demand through deduplication, increase permitted quota, use an approved alternative model, or tolerate a queue. More workers alone cannot fix that ceiling. These are simplified planning estimates. Production queueing requires checking service time variability, retries, token based limits, bursts, and end to end latency, not only averages. What metrics should drive autoscaling? 1. Queue depth and oldest job age: Queue length shows work waiting; oldest age helps detect workflows approaching a context deadline. 2. Arrival and completion rates: Measure unique valid compaction requests per minute versus successfully validated summaries per minute—not just attempted tasks. 3. Queue latency and p95 execution time: Long jobs and variable model response times change required capacity. 4. Workers' active concurrency: Track jobs executing, waiting for network I/O, and worker saturation; CPU utilization alone may be misleading for LLM API bound work. 5. Rate limit headroom: Monitor remaining request/token quota, 429 errors, and retry after signals; do not scale above a shared upstream cap. 6. Workflow impact: Track p95 end to end latency, compaction blocked handoffs, context overflow events, fallback retrieval, and successful test suite publication. 7. Costs and quality: Watch compaction spend per accepted workflow, duplicate work, stale candidate rejection, and summary validation failure rates. A scaling signal can combine queue pressure and deadline risk with a hard maximum based on vendor quota and cost policy. Step by step production design 1. Isolate the work: Give compaction its own Celery queue and a controlled worker pool so background summaries cannot consume all workers needed for security reviews or publications. 2. Deduplicate requests: Coalesce tasks keyed by tenant + workflow + source revision + role; autoscale on unique useful work rather than a flood of duplicate attempts. 3. Choose the execution model: For I/O bound API calls, modest worker concurrency may be effective; for CPU heavy local summaries, size to actual CPU and memory constraints. 4. Define minimum and maximum replicas: Keep a small warm capacity where useful, and cap scaling according to provider quotas, spend limits, and infrastructure capacity. 5. Set target capacity: Estimate service rate per worker from observed p95 workload and utilization goals; recalibrate as source document sizes change. 6. Scale up on sustained pressure: Use queued work age, arrival rate pressure, and pending deadlines; react quickly enough to protect downstream context windows. 7. Scale down conservatively: Use a cooldown or stabilization period and avoid terminating workers that own important in flight attempts. Support graceful shutdown, acknowledgments, and lease or visibility timeout behavior appropriate to the queue. 8. Make redelivery harmless: Use stable task/summary IDs and conditional activation. Two workers must never create two conflicting active summaries for the same revision. 9. Provide overload behavior: If no validated summary is ready, the next agent may use a verified older summary with a complete event tail, retrieve targeted evidence, or block safely when the prompt would overflow. 10. Test before production: Simulate bursts, quota exhaustion, worker crashes, long running jobs, stale revisions, and cross tenant contention. Queue based autoscaling example In Kubernetes, a queue driven autoscaler such as KEDA can scale Celery worker deployments based on supported broker metrics. A native Kubernetes HPA can also use custom or external metrics when correctly configured. For illustration, if there are 90 queued unique jobs and the desired pending backlog is 15 per worker , a simple queue pressure estimate would request ceil(90 / 15) = 6 replicas , subject to configured minima/maxima. This heuristic is not a guarantee of a specific wait time: queued age, service time, incoming work, and provider limits determine actual latency. Important: Queue length alone can be misleading when workers prefetch and reserve tasks. Consider ready, reserved, and active jobs, plus oldest pending age. Autoscaling based on only broker visible messages may underestimate work already assigned to workers. Prevent scaling from amplifying failures Imagine the summarization provider starts returning 429 errors. New workers increase request pressure, causing more retries and even longer queues. A robust controller should distinguish capacity shortage from upstream throttling . Enforce a global request/token rate limit shared across workers. Honor retry after responses and use bounded exponential backoff with jitter. Reduce or pause concurrency when failures indicate provider saturation. Use per tenant quotas and fair scheduling to stop one customer from exhausting all resources. Alert if successful summary throughput drops while worker count rises. LangGraph and Celery considerations In LangGraph , queue compaction at a safe node boundary and persist the source revision and required consumer. A downstream model node must confirm that its selected summary is validated and fresh enough for the task. A graph checkpoint does not itself autoscale Celery workers or guarantee completion of a queued summary. In Celery , workers can run on separately scaled deployments subscribed to a dedicated compaction queue. Configure task acknowledgment, prefetch, soft/hard time limits, and graceful termination with the workload's delivery semantics in mind. Autoscaling can cause redelivery during shutdown; therefore summary generation and activation must be idempotent and revision checked. How should success be evaluated? Compare periods with and without autoscaling, controlling for workflow mix and load. Measure p95 compaction queue wait, oldest job age, blocked agent steps, completed summaries per minute, overall workflow p95 latency, total compute/model cost, and validation quality . Include workload bursts and rate limit incidents; do not claim a speedup merely because more workers were created. Difference from earlier lessons Q200: Distributed rate limits across AI agents. Q230: When a workflow triggers compaction. Q231: Concurrency safe summary publication. Q236: Cost and latency impact of compaction. Q237: Prioritization and queue scheduling of compaction jobs. This lesson: Dynamically determines how many compaction workers should run based on load, queue age, provider quotas, and safe scale in behavior. In short Autoscale based on useful backlog and completion capacity—not CPU or queue length alone. Bound replicas by provider quotas and spend, preserve critical agent capacity, and enforce idempotent, version safe compaction so elasticity never sacrifices correctness. Further reading Celery: Optimizing worker behavior KEDA: Scalers Kubernetes: Horizontal Pod Autoscaling
How do you implement backpressure when memory-compaction demand exceeds worker capacity?
Backpressure in memory compaction means slowing, deferring, merging, or rejecting new compaction work when demand exceeds the capacity of workers or upstream language model APIs. Without it, queued summaries grow stale, workers exhaust resources, and agents can wait longer for context than they would without compaction. The goal is controlled degradation : preserve critical workflow decisions and source evidence while refusing to let an optional optimization overwhelm the system. Beginner friendly production incident An API test generation platform runs requirements, test generation, security review, and publication agents in LangGraph. Celery workers summarize old messages before expensive handoffs. A newly uploaded specification triggers a burst of workflows. Hundreds of background compaction requests enter the queue, but an external summarization model can serve only a limited number per minute. Celery retries rate limited calls, increasing the queue further. The security review agents begin timing out while waiting for summaries. The root cause is unbounded admission plus a retry feedback loop , not necessarily too few CPU cores. The solution is to bound useful compaction demand, isolate it from critical work, and let agents use a safe, verified alternative when a new summary is unavailable. Measure overload → Stop admitting redundant work → Enforce queue and quota limits → Prioritize urgent jobs → Use safe fallback → Recover gradually Step by step numerical example Assume compaction requests arrive at 50 jobs per minute , while the complete system—including model provider limits—can successfully finish only 30 per minute . 1. Net backlog growth = 50 − 30 = 20 jobs/minute , assuming no deduplication or retries. 2. After 5 minutes , the queue grows by about 100 jobs . 3. At a stable service rate of 30 jobs/minute , clearing those 100 already queued jobs would take at least 100 / 30 ≈ 3.33 minutes if no new work arrived . Under continued arrival of 50/minute, the backlog instead keeps growing. 4. Suppose admission control coalesces 15 duplicate or obsolete requests per minute and defers 10 nonurgent requests per minute. New admitted demand falls to 25 jobs/minute . 5. With a capacity of 30 jobs/minute , the backlog can now shrink by approximately 5 jobs/minute , provided the deferred requests do not immediately reenter the queue and retries are controlled. 6. A starting backlog of 100 therefore needs roughly 20 minutes to drain at the 5 job/minute net rate. If the team temporarily stops new admission, it could drain in about 3.33 minutes instead. These are illustrative steady state estimates. Actual latency depends on service time variability, burstiness, reserved tasks, token quotas, priority classes, and failures. How do you recognize compaction overload? Incoming unique jobs exceed validated completions for a sustained period. Queue depth, oldest queued job age , and p95 wait time rise. Summaries finish after their source revision is obsolete; stale candidate rejection grows. Model provider 429 responses , timeouts, and retry counts increase. Other agent tasks wait because compaction shares worker slots or API quotas. More workflows reach the next model call context limit before an acceptable summary is ready. A long queue alone is not proof of an incident: compare expected completion deadlines, actual downstream blocking, and successful throughput. Production backpressure controls 1. Bound admission: Set limits on outstanding compaction work by tenant, workflow, and queue. Admit a new task only when a capacity slot or explicit quota is available. 2. Coalesce equivalent requests: Deduplicate by tenant, workflow, source revision or event watermark, and agent role. A new revision may supersede a queued obsolete candidate, but must not erase the underlying committed events. 3. Separate urgent and background work: Give deadline sensitive pre handoff compactions priority over optional maintenance. Keep security review and publication capacity isolated. 4. Use queue age deadlines: A compaction no longer useful after the next handoff should not run indefinitely. Mark it expired and route the workflow to an approved fallback, rather than processing a stale summary. 5. Enforce global API quotas: Coordinate request and token limits across Celery workers. A per worker Celery task rate limit is not automatically a global distributed limit. 6. Limit retries: Retry transient 429 or connection failures with bounded exponential backoff, jitter, and provider supplied retry guidance where available. Do not instantly requeue every failure. 7. Keep prefetch controlled: Long running Celery tasks reserved far ahead can conceal backlog and starve urgent work. Tune worker prefetch multiplier and routing for task duration and broker behavior. 8. Use per tenant fairness: Prevent one tenant's large ingestion burst from consuming all compaction capacity. 9. Protect downstream nodes: Before a model call, verify the context still fits and that its source facts, approvals, and version are valid. 10. Shed optional work first: Skip nonessential compaction or use targeted retrieval. Never silently skip authorization checks or treat a pending approval as passed. What should an agent do when compaction is delayed? A useful fallback hierarchy is: Reuse a validated summary plus a complete authorized event tail when it fits the model's safe context budget. Retrieve only the required source sections and build a role specific context where that covers the task safely. Use a smaller, prevalidated working view if it retains all mandatory constraints. Delay or block the affected model call if no safe context can be assembled. Do not use a partial, timed out, or unvalidated summary just to keep the queue moving. The underlying workflow can remain resumable through persisted state. Example: preventing a retry storm Imagine 40 Celery compaction tasks simultaneously encounter an upstream quota limit. Without coordinated throttling, each immediately retries, creating another 40 requests, and potentially more timeouts. Use a shared limiter or quota scheduler to admit only calls supported by remaining provider capacity, and return deferred tasks to a bounded delayed work mechanism . Where possible, honor Retry After, add jitter, and cap attempts. Investigate sustained errors instead of retrying indefinitely. Do not assume ordinary Celery task rate limit coordinates a quota across every worker: documented task rate limits apply per worker instance. A broker visible queue can also undercount work already reserved through prefetch. LangGraph and Celery implementation In LangGraph , trigger compaction at stable node or checkpoint boundaries, but keep the full authorized source evidence and durable state separate from the model facing summary. A node needing a compact context should check whether a validated summary at the required revision is available and then select a safe fallback or pause. Checkpoint persistence does not itself enforce backpressure. With Celery , isolate compaction queues, bound worker concurrency and prefetch, track queue delay separately from execution time, and put admission decisions ahead of task creation when possible. Use stable task IDs or logical deduplication keys. Compaction results must be idempotent and activated through version aware comparison so late workers cannot overwrite newer summaries. For example, an admission controller can follow this simplified logic: Actual implementation needs atomic admission accounting, expiry handling, and recovery of abandoned reservations. A check then enqueue sequence without concurrency protection can still over admit bursts. Trade offs and metrics Hard queue limits protect capacity but can delay useful compactions; attach explicit workflow safe fallback handling. Coalescing saves work but requires careful treatment of source versions and different role specific contexts. Priority queues help critical handoffs but need aging or fair share controls to avoid starving background tasks. Throttling reduces errors but increases queue wait, so context deadlines must remain visible. Additional workers help only when the true bottleneck is worker capacity, not the shared LLM quota. Track admitted vs. deferred jobs, unique arrival/completion rates, oldest queue age, p95 wait, 429s, retry amplification, stale candidates, blocked workflow steps, safe fallback success, context overflow, and accepted task quality . How does this differ from earlier questions? Q181: General backpressure for communication and task flow across agents. Q237: Scheduling and prioritizing compaction work efficiently. Q238: Adding or removing compaction worker capacity. This lesson: Controlling incoming demand and overload behavior when capacity cannot keep up, including admission limits, coalescing, retry suppression, and safe context fallback. In short When compaction demand exceeds sustainable capacity, regulate arrivals rather than just adding workers. Bound queues, deduplicate superseded work, honor global model quotas, limit retries, and preserve a verified path for agents to continue—or safely pause—without losing critical context. Further reading Celery: Optimizing Celery: Task rate limits and retries LangGraph: Add and manage memory
How do you prioritize critical memory-compaction requests during system overload?
During system overload, prioritize memory compaction requests by how urgently a validated summary is needed for a safe next agent step—not simply by which job entered the queue first. The coordinator must preserve capacity for critical agent work, avoid tenant starvation, and never promote an incomplete summary to satisfy a deadline. This lesson focuses on priority assignment and fairness under an already overloaded system . Queue admission and backpressure are separate mechanisms. Beginner friendly production incident A LangGraph pipeline has requirements, test generation, security review, and publication agents. Celery compaction workers receive a burst of summary requests after an API specification is uploaded. Two jobs are waiting: Job A: The security reviewer cannot proceed because the next prompt would exceed its safe context budget in 20 seconds. The required OAuth scopes and pending approval must be preserved. Job B: A completed workflow requests an optional historical summary for later analytics; it has no immediate consumer. A FIFO queue might process Job B first. Priority aware scheduling favors Job A, but only if there is enough time to produce and validate the summary; otherwise, the reviewer uses a verified fallback or pauses safely. Classify next step criticality → Calculate time to deadline → Check context safety → Reserve fair capacity → Dispatch → Validate → Activate or fall back Step by step numerical example Suppose one available compaction worker can finish and validate a job in approximately 10 seconds . Four jobs arrive together: 1. A — Security review handoff: Needed in 15 seconds ; otherwise the prompt would exceed its approved budget. 2. B — Test generator handoff: Needed in 40 seconds ; safe verified context is available for now. 3. C — Routine analytics: Needed in 10 minutes , with no blocked workflow. 4. D — Another tenant's security review handoff: Needed in 25 seconds . A possible order is A → D → B → C . A completes at approximately 10 seconds , before its 15 second deadline. D completes at approximately 20 seconds , before its 25 second deadline. B completes at approximately 30 seconds , before its 40 second deadline. C completes at approximately 40 seconds , well before its 10 minute deadline. This works under the idealized fixed duration assumptions. In real deployments, task duration varies, and a job's deadline includes queue wait + model execution + validation + activation . An already running Celery task is not safely preempted just because a higher priority job arrives. How should a priority score be designed? Use a small number of explainable signals rather than arbitrary labels: 1. Deadline slack: slack = time remaining − estimated queue and execution time. Lower positive slack means the job is more urgent; negative slack may mean the required summary cannot be delivered on time. 2. Blocked workflow: Give additional priority when a critical agent is actually waiting and has no safe alternative context. 3. Risk of context overflow: Compaction becomes urgent when the projected next model input exceeds its verified usable budget. 4. Downstream importance: A pending authorization review or required human approval handoff matters more than an optional summary for offline reporting. 5. Expected reuse: A valid summary needed by several immediate next nodes can provide more value, provided it does not starve urgent one off work. 6. Source readiness: Prefer jobs whose source revision is stable enough that the summary is likely to remain valid; avoid repeatedly prioritizing jobs invalidated by active writes. 7. Tenant fairness: Apply per tenant quotas, weighted fair sharing, and aging so a noisy tenant cannot monopolize the queue. Do not confuse priority with permission to bypass validation. Security critical summaries often require stronger validation, which must be included in the time estimate. Suggested priority classes Critical: A safe model call cannot proceed without a validated summary or another safe recovery path, and the deadline is close. High: A summary is required at an upcoming handoff, but some headroom remains. Normal: Beneficial compaction for running workflows without immediate context pressure. Background: Maintenance, optional analytics, and speculative context precomputation. Use deadline ordering or aging within classes and reserve a limited share of service capacity for lower priorities to prevent indefinite starvation. Class names are operational conventions, not inherent model properties. What if a deadline cannot be met? Imagine Job A has 5 seconds before a required handoff, but generation and validation normally need 10 seconds . Making it the highest priority does not create extra capacity. The coordinator should: Use the last validated summary plus a complete authorized event tail , if it fits. Retrieve only the necessary source requirements and approval evidence, if that fits the model budget. Delay or pause the affected node at a safe checkpoint when no complete trustworthy context is available. Avoid using a partial summary, dropping important facts, or treating pending approval as granted. It may still be worthwhile to schedule a later compaction for future calls, but not to claim that it met an impossible deadline. Fairness under sustained overload Strict priority can starve routine tasks indefinitely. For example, if urgent work continuously fills the queue, low priority maintenance may never run. A fair scheduler can: Reserve a bounded portion of worker/API capacity for noncritical work. Increase a task's effective priority as its waiting age grows. Cap each tenant's concurrent compaction jobs. Coalesce equivalent jobs so multiple agents do not consume separate capacity for identical revisions. Expire jobs after their intended consumer has passed, finished, or moved to a new source revision. Fairness must not override hard security gates or allow another tenant's data into an agent's compact context. Celery implementation details Use dedicated queues for urgent versus background compaction, with explicitly allocated workers or broker supported priorities where available. Celery priority behavior depends on the broker and is not identical across Redis and RabbitMQ configurations. A high priority message may still wait if every worker has already reserved long running low priority tasks. Tune prefetch according to execution time; isolating urgent workers may help more than relying only on broker priority. Tasks should carry workflow ID, tenant ID, source revision, priority class, deadline, and logical deduplication key. Priorities should be reassessed when a job's source revision becomes stale or its consumer no longer needs the summary. A stale compaction task must not overwrite newer validated context. LangGraph implementation details At each handoff, calculate whether the next node can run using validated memory within its context budget. If not, request a compaction with an explicit consumer and deadline, then persist the workflow state. When compaction finishes, verify the candidate's version and content before resuming. LangGraph checkpoints help a workflow resume; they do not automatically prioritize Celery tasks or provide global queue fairness. Priority control belongs in the scheduling, admission, and worker architecture. Metrics to monitor Track deadline miss rate by class, p95 queue age, critical agent blocked time, percentage of critical jobs meeting validation gates, priority inversion events, fairness by tenant, starvation age, unnecessary compactions, stale candidates, and downstream task correctness . An apparent speedup is not acceptable if skipped validation leads to missing authorization tests. Analyze impact on end to end accepted workflow latency and cost, not merely worker throughput. Difference from nearby questions Q198: Prevents starvation in general multi agent scheduling. Q237: Improves overall compaction scheduling and bottlenecks. Q238: Scales worker capacity. Q239: Limits incoming demand and establishes safe overload behavior. This lesson: Selects which admitted compaction requests receive scarce capacity first , using deadlines, context urgency, workflow importance, and tenant fairness. In short Prioritize compaction based on deadline slack and the next agent's need for a validated context, while reserving fair capacity and enforcing safety gates. If a critical deadline cannot be met, use verified fallback context or pause the workflow—never ship an unvalidated summary. Further reading Celery: Routing Tasks Celery: Optimizing LangGraph: Persistence