How do you reconcile orphaned and retried memory-compaction jobs after an AI-agent controller policy cutover?

Reconciliation is the process of accounting for every compaction job affected by a policy-version cutover and safely deciding whether it has already completed, must be replayed under the new policy, should be cancelled, or needs human review. Fencing (Question 257) stops obsolete workers from *writing*; it does not guarantee that necessary work gets completed. The complementary goal is no lost required work and no duplicate memory mutation.

Why jobs become orphaned

A controller upgrade may leave Celery messages waiting, reserved or running, while a LangGraph thread has already checkpointed a different state. A worker crash between a storage commit and an acknowledgement creates uncertainty: a broker may redeliver a message even though the memory was updated. Conversely, an early acknowledgement followed by a crash may leave work unperformed. Celery acknowledgement and worker-loss behavior depend on configuration and failure mode; neither the queue nor a result backend alone is authoritative proof of successful mutation.

Beginner-friendly numerical incident

A production deployment replaces policy v7 with v8 while 600 compaction jobs exist: 400 queued, 120 reserved and 80 running. From a durable job ledger, after a settling window the reconciler finds:

• 420 already committed with a valid memory revision and recorded output.
• 90 pending and safe to schedule under policy v8.
• 45 attempted by fenced v7 workers, correctly rejected by the storage guard.
• 25 broker redeliveries whose original logical operations are already among the 420 commits.
• 20 ambiguous because a downstream vector-index update or acknowledgement has an unknown outcome.

The five mutually exclusive *reconciliation classifications* total 600. The 25 redelivered deliveries must not create another 25 mutations; use the logical operation ID to return the previously recorded outcome. Replan the 90 pending and 45 fenced cases using current checkpoints (135 candidates), but only if still needed. Quarantine the 20 ambiguous cases pending authoritative inspection. Illustrative result: 420 verified completions + 115 subsequently verified replanned completions + 20 superseded/cancelled candidates + 20 resolved ambiguities + 25 duplicate deliveries accounted for = 600 accounted-for original jobs. A duplicate delivery being accounted for is not a unique successful compaction.

Authoritative reconciliation ledger

Track one row per logical compaction request, separate from delivery attempts. Fields: logical_operation_id, tenant/workflow/memory scope, original and current policy version, controller fencing generation, input checkpoint ID and memory revision, request timestamp, attempt IDs, immutable candidate hash, committed revision, external-side-effect status, terminal disposition and operator/audit IDs. Deduplicate on the logical operation, not a Celery task delivery ID.

Use states such as REQUESTED → CLAIMED → COMMIT_VERIFIED → DONE, with REPLAN, SUPERSEDED, QUARANTINED and CANCELLED as distinct dispositions. A timeout is a signal to inspect, not evidence that a transaction failed. The ledger must be transactional and access-controlled.

Recovery procedure

1. Create a stable cutover boundary: record policy versions, fence generation, active queues, durable workflow/checkpoint references, and ledger high-watermark. Continue recording new arrivals independently.
2. Join evidence across layers: correlate queue delivery attempts, worker events, LangGraph checkpoint transitions, transactional memory revisions, idempotency records, outbox events and downstream-index receipts by logical operation ID. Avoid assuming timestamps from different hosts establish total ordering.
3. Prove committed effects: prefer an authoritative database revision and matching content hash over a worker 'success' log. For external effects, use an outbox/receipt and reconcile discrepancies.
4. Classify each logical operation: DONE if the intended effect exists; REPLAN if required and absent; SUPERSEDED if a newer compaction makes it unnecessary; QUARANTINED if the result cannot be established; CANCELLED if safety rules prohibit it.
5. Replan, do not blindly replay: obtain the current checkpoint and policy, recompute eligibility, validate protected canonical facts, issue a new attempt that references the same business operation or a documented replacement operation, and attach the current fence. A stale candidate computed against the old memory revision must not be committed.
6. Commit idempotently: atomically mark the operation and memory revision; use a transactional outbox to request downstream work. Consumers deduplicate outbox event IDs. An external vector index lacking transactions may require versioned writes, index reconciliation, or quarantine.
7. Measure closure: require every original logical operation to have one terminal disposition, no accepted stale-generation commits, no double-commits, and no missing canonical handoff facts; audit any operator overrides. Do not silently mark ambiguous cases successful.

Example recovery decisions

• A v7 job reports success, but the committed revision is absent: inspect and replan after checking downstream receipts; don't trust the status message alone.
• A redelivered job's logical_operation_id has a verified commit: acknowledge as duplicate, return the stored result; do not compact again.
• A job was fenced after its checkpoint became stale: refresh context and re-evaluate under v8, rather than reusing the obsolete payload.
• The database commit exists but search-index receipt is missing: reconcile or replay the idempotent index update, not the memory mutation.
• Required context appears to have been deleted: quarantine/escalate and recover from an authoritative snapshot; never auto-approve a guessed summary.

Operational monitoring and trade-offs

Monitor the number and oldest age of unresolved logical jobs, ambiguous-result ratio, replay-to-commit ratio, duplicate delivery count, accepted stale writes (must be zero), outbox lag, reconciliation throughput and canonical-context verification failures. For example, with 20 ambiguous of 600, the ambiguity rate is 3.33%; use a predefined incident threshold and investigate rather than hiding cases by declaring success.

A strong ledger and outbox add storage, transactional work and integration complexity. Conservative quarantine reduces corruption risk but raises latency and operator load. Automatic replay reduces backlog but increases cost and may repeat external side effects unless idempotency is enforced end-to-end. Exactly-once execution across the whole distributed stack is not a safe assumption; design for repeatable attempts with effectively once-applied, verifiable business effects.

Difference from earlier lessons

Questions 255–256 cover recalibration versus retirement/replacement; Question 257 prevents conflicting policy generations from committing; Question 258 accounts for every outstanding logical operation and restores legitimate unfinished work. Shared-memory schema migration lessons address data-format changes, not queue/ledger reconciliation after policy cutover.

Sources and further reading

• Celery, Tasks (acknowledgements, redelivery, idempotency): https://docs.celeryq.dev/en/stable/userguide/tasks.html
• LangGraph, durable execution and replay: https://docs.langchain.com/oss/python/langgraph/durable-execution
• AWS Prescriptive Guidance, transactional outbox: https://docs.aws.amazon.com/prescriptive-guidance/latest/cloud-design-patterns/transactional-outbox.html
• PostgreSQL transaction isolation: https://www.postgresql.org/docs/current/transaction-iso.html