How do you safely restore automated AI-agent memory tombstone garbage collection after a resurrection incident?

After a stale-memory resurrection incident, automated tombstone garbage collection (GC) must remain disabled until independent evidence demonstrates that the offending replication, lineage, and replay paths are repaired. Re-enabling a timer or resetting the failed alarm is insufficient. Restore in narrowly scoped cohorts with explicit hold states, pre-commit eligibility checks, independent observation, and automatic stop conditions.

Relationship to earlier lessons

• Q262 explains how delayed replication resurrects compacted memory.
• Q263 establishes tombstone GC eligibility and replica-watermark requirements.
• Q264 handles the *active resurrection incident*: detection, containment, forensics, and data repair.
• **Q265 explains safe *return to automated GC* after that repair**, including reliability gates, operating limits, rollback, and sustained verification. Unlike general production incident response (Q213) or recurrence prevention (Q215), it is specific to the distributed memory-invalidation controller and its data-integrity invariants.

Beginner-friendly numerical production example

A three-region LangGraph/Celery platform serves 10,000 workflows. An incorrect GC policy previously reclaimed invalidation metadata for 12,000 summaries while region C had not acknowledged 60 of them. It allowed 80 old summaries to resurface, which were repaired under Q264.

The revised controller sees 12,000 GC candidates but checks them against fresh, independently verified evidence:

• 11,880 candidates have contiguous applied replica watermarks, validated downstream retrieval/index revisions, replay/backup protections, and no outstanding privacy or audit hold.
• 90 have an offline or unverified consumer.
• 30 have unresolved indexing/checkpoint lineage.
• Totals: 11,880 + 90 + 30 = 12,000, and 11,880 / 12,000 = 99% eligible *by the illustrative application-level gates*; the other 120 must remain on hold.
• The 11,880 still require storage-engine-specific safeguards and any applicable retention/erasure policy. A 99% eligibility rate does not justify deleting the other 1%.

Start with 100 approved records in one low-risk tenant cohort, then 500, then at most 2,000 per controlled interval if each previous cohort passes checks. These are example rollout values, not universally safe rates. Suppose the 500-record stage produces 2 stale-retrieval observations from an independent read probe. The safety gate requires zero; immediately stop further GC and quarantine affected scopes, regardless of successful deletion throughput. Do not interpret the two observations as proof that GC caused them until causality is investigated.

The safe restoration sequence

1. Record a formal remediation baseline. Pin the root-cause findings, corrected GC code and config hashes, data repair manifest, region membership and fencing epochs, authoritative per-memory revision pointers, index rebuild receipts, replay blockers, policy owner and rollback version.
2. Require an independent health gate. Use signals *not produced solely by the GC controller*: replicated state audits, contiguous per-partition apply watermarks, replica membership attestations, queue/outbox gap checks, authoritative lineage comparisons and retrieval probes against known invalidated revisions. If any critical signal is stale or missing, enter UNKNOWN/HOLD, not PASS.
3. Test previously failing conditions in staging. Rejoin an offline replica carrying obsolete memory; inject out-of-order replication, Celery redeliveries, paused LangGraph threads, index refresh, expired lease, old snapshot restore and incomplete repair. Require zero accepted invalid revisions, zero invalidated summaries returned, and full accounting of candidate decisions.
4. Run read-only shadow eligibility. The new controller computes candidate GC decisions and reason codes but deletes nothing. Compare its decisions with an independent validator and manual incident samples; no unexplained mismatches may pass.
5. Permit small, reversible rollout boundaries. GC metadata deletion may be irreversible, so *reversibility* means pausing further deletion while maintaining sufficient independent revision authority and protected snapshots or replay controls—not restoring unsafe old tombstone state. Scope by tenant, partition and batch size, with rate limits and a minimum interval between cohorts.
6. Revalidate immediately before effect. Compare the candidate's version, applied watermark, membership version, hold state, retention permissions and authoritative revision under a consistent transactional decision or equivalent guarded protocol. An earlier PASS must not be reused after a new replica joins or a consumer falls behind.
7. Monitor during and after. Check read-path validity, not just worker success: invalidated memory retrieved, outdated summaries served, accepted stale writes, index revision mismatch, ledger gaps, repair lag, per-tenant impact and GC backlog. Escalate any safety breach. Keep a named human approver for expansion and a tested kill switch.
8. Close only after sustained evidence. Include delayed retries, offline-node returns and the longest relevant compaction/replay cycles. Publish audit records and corrective actions; avoid claiming zero future risk based on finite tests.

Why an ordinary threshold is not enough

Consider a GC candidate with invalidation log offset 840. Region A's contiguous verified watermark is 920, B's 905, but region C's is 815. The minimum is 815, so this candidate must not be collected. If C advances to 870, the watermark condition passes; nevertheless, a pending replay consumer at 830 may still require the marker or an independent authoritative revision gate. Check *every eligible path*, not just regions. No heartbeat or timeout substitutes for acknowledged progress.

A practical decision is:

• PASS only if all mandatory evidence is fresh, coherent and independently validated; the candidate's scope has no blockers.
• HOLD if a known consumer/replica or policy requirement blocks cleanup.
• UNKNOWN if required telemetry is unavailable, untrustworthy, or contradictory. No destructive GC.
• STOP/INCIDENT if an invalid revision is served/accepted, lineage diverges, or an explicit safety invariant is broken.

Production failure cases, trade-offs and common mistakes

• Premature reopening: recovery dashboards show healthy regions but external indexes still contain invalid entries. Use independent retrieval probes and provenance.
• Metrics from the same controller: a faulty reconciler can mark its own destructive action safe. Cross-check against canonical storage and independently collected replication evidence.
• Stale green decision: one region falls behind between scan and deletion. Recheck membership and watermark at commit; fail closed on mismatch.
• Fast batch expansion: a new defect spreads across tenants before delayed-read failures appear. Require observation intervals grounded in actual retry and data-lag tails.
• Excessively conservative global freeze: safe scopes may accumulate storage pressure while one replica is offline. Allow partition-scoped progress only with independently enforced isolation and audited exclusions.
• Engine-specific caveat: Apache Cassandra's actual tombstone lifecycle is governed by compaction, repair and gc_grace_seconds. An application-level safety certificate does not override those engine rules or make arbitrary manual SSTable deletion safe.
• Privacy and retention: preserve minimal non-content validity metadata where appropriate, but do not keep legally erased payloads to make debugging easier. Document any retention and security exceptions.
• Overclaiming success: zero failures in a finite probe sample cannot demonstrate a zero underlying probability of future resurrection.

Operational scorecard and ownership

Record eligible/held/unknown candidate counts, oldest blocked marker, contiguous replica and consumer watermarks, independent validator disagreement count, stale read/write acceptance (target zero), index revision mismatches, quarantine backlog, GC batch duration, storage recovered, and rollout-cohort status. Every override must identify approver, rationale, expiry, affected scopes and revalidation evidence. Trigger a stop if any invalid revision is observed in serving paths or if mandatory health telemetry becomes unknown.

Takeaway: repair is a prerequisite, not a green light. Independent verification → shadow decisions → small canary GC → read-path proof → controlled expansion, with fail-closed operation if evidence disappears.

References

• Apache Cassandra, Tombstones: https://cassandra.apache.org/doc/stable/cassandra/managing/operating/compaction/tombstones.html
• Apache Cassandra, Repair: https://cassandra.apache.org/doc/stable/cassandra/managing/operating/repair.html
• Google SRE Workbook, Postmortem culture and corrective actions: https://sre.google/workbook/postmortem-culture/
• LangGraph persistence: https://docs.langchain.com/oss/python/langgraph/persistence
• Celery task acknowledgements and retries: https://docs.celeryq.dev/en/stable/userguide/tasks.html