How do you recover from a failed memory-compaction operation in a multi-agent workflow?

Recovering from failed memory compaction means keeping the last trustworthy working context available, preventing partial summaries from becoming active, and rebuilding a valid compact context without losing agent updates. Compaction is an optimization—not an authoritative write of the task's underlying facts—so a compaction failure should normally leave durable workflow events, approvals, and artifacts intact.

Beginner-friendly production incident

A four-agent API-test generation workflow uses a requirements agent, test generator, security reviewer, and publisher. After 30 minutes of work, a compactor starts shortening a long conversation for the next model call.

The summarization model times out after generating only part of the summary. That partial output omits an unresolved OAuth scope issue. If the publisher reads the incomplete summary as authoritative, it might publish tests without the required authorization checks.

Correct recovery: Reject the incomplete candidate, retain the last verified active context, inspect the source revision, retry safely when appropriate, and prevent the publisher from acting until its required facts and approvals are verified.

Detect failure → Isolate candidate → Preserve canonical state → Classify cause → Retry or fallback → Revalidate → Atomically activate → Monitor

Step-by-step numerical example

Suppose the canonical event history contains 24,000 tokens and the last validated summary covers events 1–150. Agents have since committed events 151–175.

1. A compaction worker takes a snapshot through event 175 and attempts a new 6,000-token summary.
2. After 4 seconds, the summarization request times out. The incomplete candidate is not promoted.
3. While the worker is failing, agents commit 5 more events, advancing the canonical event watermark to 180.
4. A retry using the old snapshot still covers only events 1–175; it cannot silently replace context that includes events 176–180.
5. The safe immediate context is the last validated summary through 150 plus durable events 151–180, assembled with revision-aware reads.
6. The compactor generates a new candidate through 180, checks required constraints and pending reviews, then publishes it with a conditional revision check. If the source advances again, retry or retain an uncompacted tail.
7. The system loses 0 committed events. The temporary token and latency cost can increase, but correctness is preserved.

This example assumes durable, gap-free event storage and consistent read semantics. A fallback that itself exceeds the model's token limit must use targeted retrieval, a smaller safe window, or block that model call rather than discard important facts.

Distinguish common failure types

• Model timeout or rate limit: The summarizer never completes. Retry with bounded exponential backoff and jitter when transient, without changing the active summary.
• Malformed or incomplete output: JSON, schema, or required fields fail validation. Reject and regenerate with a stricter schema or smaller source partitions.
• Semantic validation failure: The candidate omits a critical constraint, changes approval status, or invents a source reference. Quarantine it for repair or review; do not blindly repeat an identical failing transformation indefinitely.
• Concurrent-state conflict: Another agent writes new events before activation. Compare-and-swap rejects the stale candidate; reconstruct from the newest revision or a stable watermark.
• Database or storage failure: A candidate may have been written but its active pointer may not have changed. Read the committed pointer and validation status to determine whether the operation completed.
• Worker crash/redelivery: Celery may rerun a task after a crash. Use stable attempt IDs, candidate IDs, and idempotent writes to avoid duplicate promotion or corrupt state.
• Downstream regression discovered later: Revert the active summary pointer to the last validated version and append the uncompacted event tail, then investigate affected outputs and side effects.

How do you make compaction crash-safe?

1. Retain an immutable source of truth: Store original facts, event IDs, tool outcomes, source references, and approval decisions in durable storage under the proper retention policy.
2. Version every candidate: Record workflow ID, tenant ID, source watermark, compaction algorithm/prompt version, candidate ID, and creation attempt.
3. Use staged activation: Save candidate as PENDING, validate it, then mark VALIDATED; only a validated candidate may become the active summary.
4. Activate atomically: Use a transaction or revision-checked compare-and-swap for the active-summary pointer. A failed or stale attempt must not overwrite newer context.
5. Implement bounded retries: Define maximum attempts, backoff, timeouts, and a dead-letter or review path for persistent errors.
6. Support safe fallback: Assemble context from a previous verified summary plus uncompacted events, or retrieve source evidence on demand.
7. Protect critical actions: Before publishing or writing externally, verify authorization facts and approval states against durable sources, not only compressed text.
8. Reconcile uncertain outcomes: After a timeout during database activation, read the authoritative pointer before retrying; do not assume a timed-out operation definitely failed.
9. Keep tenant isolation: Every candidate, replay, and fallback must preserve tenant scope and retrieval permissions.
10. Test recovery paths: Inject model timeouts, crashes before/after pointer updates, stale revisions, corrupt summaries, and missing source records.

Example of an atomic activation rule

The compactor produces candidate s180 based on source revision 180. An illustrative conditional update is:

SQL
UPDATE workflow_memory
SET active_summary_id = :candidate_id,
    summary_watermark = :source_revision
WHERE workflow_id = :workflow_id
  AND tenant_id = :tenant_id
  AND canonical_revision = :source_revision;

This is only one possible design. The candidate must already be durably stored and validated, and an application-level transaction must preserve its relationship to the pointer. Zero rows updated means the expected revision did not match or the target is unavailable; do not force the promotion.

For a watermark-plus-tail design, readers may safely use a validated summary through an earlier revision if they also include every authorized event after that watermark through a consistent read boundary.

When should retries stop?

Retry transient errors, such as connection timeouts, with limits. Stop automatic promotion on safety or semantic failures.

For example, if three attempts omit the same mandatory approval, escalating for review is safer than accepting the shortest output. A fallback should be observable and bounded: continuing with unlimited raw history could repeatedly hit the context limit or increase cost unexpectedly.

LangGraph and Celery example

In LangGraph, keep checkpoints and durable facts separate from the model-facing summary. Save validated summaries in versioned state or a store, and use a graph node to choose the active context plus unsummarized tail. Test restoring a checkpoint after a compaction crash; a checkpoint is not a guarantee that an externally stored summary is still valid.

With Celery, give each compaction job a stable key such as tenant_id + workflow_id + source_revision. Mark results idempotently, use retry/backoff for transient failures, and always check the source revision before activation. A lease can reduce duplicate compaction work, but cannot replace transactional correctness if it expires or a worker is redelivered.

Metrics and trade-offs

Track compaction failure rate, retries per candidate, stale-candidate rejections, recovery latency, fallback context length, context-overflow errors, missing-event incidents, and downstream task regressions.

Keeping previous summaries and raw events uses more storage, and safe fallback can temporarily increase tokens and latency. Those costs are usually preferable to lost decisions or unsafe publications. Avoid retaining sensitive raw content longer than the approved retention policy.

Difference from earlier lessons

• Q227: Implements compaction.
• Q228: Defines information-preservation checks.
• Q229: Evaluates quality.
• Q230: Chooses trigger timing.
• Q231: Coordinates concurrent updates and atomic activation.
• This lesson: Recovers from timeouts, invalid candidates, worker crashes, ambiguous commits, and post-promotion defects without dropping committed agent work.

In short

Treat compaction output as a replaceable, validated cache of durable evidence. On failure, leave the active context unchanged, keep all newer events, retry only when safe, and promote a new summary through a version-checked atomic operation.