How do you verify a memory-compaction controller cutover is complete and safe before decommissioning old AI-agent workers?

A safe controller cutover is complete only when both control-plane authority and data-plane outcomes are verified: new decisions use the authorized policy, old workers can no longer commit, every pre-cutover logical job has a justified disposition, and delayed context-quality regressions remain within approved limits. A green deployment dashboard or empty Celery queue alone does not establish this.

Why this is the next step

Question 256 decides whether to replace a controller; 257 introduces writer fencing; 258 reconciles orphaned/retried jobs. Question 259 defines the evidence-based release exit gate that decides whether to remove the old worker fleet and its rollback infrastructure. Unlike Question 245's general rollout guidance, this checks *post-cutover closure of stateful, asynchronous, multi-agent work*.

Beginner-friendly numerical production example

A LangGraph + Celery platform replaces memory-compaction policy v7 with v8 across 10,000 workflows. The release team predefines these acceptance gates:

• Authority: 100% of new mutations carry the current fencing generation; zero accepted writes from v7.
• Reconciliation: all 600 pre-cutover logical compaction jobs have a terminal, independently evidenced disposition; zero unresolved ambiguous cases.
• Quality: zero confirmed loss of protected canonical facts in an audited sample and in automated invariant checks. This is a safety gate, not proof that the true underlying risk is zero.
• Service: 99% of new compactions finish within 120 seconds; error-budget burn and manual intervention remain below agreed thresholds.
• Observation: at least one full workload/compaction cycle, including scheduled retries and downstream indexing, has elapsed with no blocking regression. The actual observation period must be based on system-specific tail latencies, not a universal number.

On day one, the dashboard shows 9,940 / 10,000 (99.4%) new compactions under 120 seconds, but only 598 / 600 (99.67%) old operations have final dispositions. Two remain ambiguous, including one index update that may have happened after its acknowledgement was lost. Do not decommission yet. Once the remaining two are investigated and closed with durable receipts, the 600/600 closure gate passes. Later, a targeted audit examines 500 completed workflows and finds 0 missing protected facts: that is encouraging, but with zero observed failures in 500 independent representative trials, the rough 95% 'rule of three' upper bound is 3/500 = 0.6%, not zero risk. Independence and representativeness are assumptions; correlated workflows make this bound optimistic. For severe failures, use stronger invariants and longer observation.

Evidence required before the final switch

1. Freeze a cutover manifest: record release ID, old/new policy hashes, scope/tenant cohorts, fencing generations, queue names, deployment IDs, rollback operator, and authoritative snapshot/checkpoint references.
2. Verify authority at the actual write boundary: query transactional memory revisions, commit audit events and generation checks; do not rely only on load-balancer traffic or worker liveness. Try a controlled stale-generation write and confirm rejection.
3. Prove all pre-cutover work accounted for: compare a fixed ledger watermark with terminal dispositions (DONE, SUPERSEDED, CANCELLED, QUARANTINED resolved, or REPLAN subsequently closed). Exclude redeliveries from the count of distinct logical jobs; never convert unknown outcomes to success.
4. Inspect dependency lag: verify outbox and downstream vector-index receipts, delayed Celery retries, scheduled tasks, LangGraph pending checkpoint writes and long-running human-paused workflows. A zero queue depth can coexist with unacknowledged or lost side effects.
5. Evaluate new-policy operation: segment by tenant, workflow type, provider, context length and time of day. Compare completion p95/p99 latency, unnecessary compactions, rate-limit events, cost, handoff correctness, and accepted stale writes. Canary/control cohorts should be isolated or contamination explicitly noted.
6. Run fault injection: late v7 completion, worker death after a memory commit but before acknowledgement, duplicate delivery, expired lease, disconnected worker, outdated checkpoint, delayed index receipt, and rollback request. Require invariants and audit trails to hold.
7. Obtain explicit sign-off: service owner approves closure evidence; safety/data owner approves protected-context checks; on-call owner confirms runbook and alert routing. Store the evidence with a release ID and expiry/retention policy.

Safe decommission procedure

• Stop new dispatch to v7 queues and disable v7 schedulers; record where messages and periodic tasks could still originate.
• Drain or cancel old deliveries *under existing fencing*, preserving ledger evidence. A revoke is not guaranteed to stop work already executing; storage-side fencing remains active.
• Remove v7 writer credentials or permissions only after validating that no other legitimate workflow depends on them. Use staged shutdown and check for residual writes or retry storms.
• Keep old policy artifacts and checkpoints read-only for audit/recovery. A rollback to old logic must issue a newer fencing generation rather than restoring an obsolete authority token.
• Monitor after decommission for unexpected v7 attempts, missing indexed records and delayed quality regressions. Automate rollback/pause triggers where safe, but require investigation before making ambiguous work authoritative.

Failure cases and trade-offs

• False green from queue depth: queue empty while two external effects are uncertain. Require end-to-end receipts and reconciliation closure.
• Success metrics hide quality loss: faster compaction but missing requirements in future agent turns. Validate final task outputs against protected canonical facts and inspect delayed failures.
• Over-short observation window: nightly workflows and long retries were absent from canary data. Select window by task tail duration and workload periodicity; extend if samples are not representative.
• Aggregate percentiles mask tenant impact: overall 99% target passes but one small tenant has high failure rates. Apply per-cohort minimums and risk-weighted review.
• Permanent old fleet for safety: retaining legacy writers increases attack surface and split-brain risk; decommission code and credentials while retaining immutable evidence and safe logical rollback capability.
• Rare-harm uncertainty: zero sampled errors does not establish impossibility. Favor hard state invariants, adversarial testing, and ongoing monitors over a single pass/fail confidence claim.

What to monitor continuously

Old-generation commit acceptance (zero), rejected stale attempts, unresolved ledger count and age, pre-cutover reconciliation percentage, unexpected retry/backlog spikes, index-outbox discrepancy age, p95/p99 time-to-safe-compaction, protected-fact loss, operator overrides, and rollback readiness. Treat any accepted stale write or confirmed canonical-fact loss as a stop-the-line event.

References

• Google SRE Workbook, *Canarying Releases*: https://sre.google/workbook/canarying-releases/
• Celery, *Tasks* (acknowledgements and retries): https://docs.celeryq.dev/en/stable/userguide/tasks.html
• Celery, *Workers* (shutdown and revokes): https://docs.celeryq.dev/en/stable/userguide/workers.html
• LangGraph, *Durable execution*: https://docs.langchain.com/oss/python/langgraph/durable-execution
• NIST AI Risk Management Framework: https://www.nist.gov/itl/ai-risk-management-framework