When should you retire or replace an automated memory-compaction remediation policy instead of recalibrating it?

Retire or replace an automated memory-compaction remediation policy when its underlying decision logic or safety assumptions no longer hold, not merely because one threshold needs adjustment. Recalibration (Question 255) changes parameters within a validated policy. Replacement changes the policy structure, evidence inputs, or permitted actions. Retirement disables unsafe automation in favor of a controlled fallback.

How to decide

• Recalibrate when incident types remain familiar, causal signals are trustworthy, and out-of-sample replay shows a bounded threshold change restores performance without harming memory integrity.
• Replace when new model-provider behavior, tenant isolation requirements, or failure modes defeat the existing logic: no viable parameter setting meets both recovery and safety targets.
• Retire temporarily when telemetry cannot prove provenance, audit logs are missing, compaction corrupts canonical handoff information, or no candidate policy passes safety checks. Use conservative fixed schedules or human approval while rebuilding evidence.
• Apply predeclared criteria and a human review for high-impact changes; do not let the degraded controller approve its own replacement.

Beginner-friendly production incident

A LangGraph + Celery multi-agent service compacts old context during provider throttling. The current policy triggers compaction when queue age exceeds 30 seconds. After a provider routing change, queue age is no longer a good indicator of token pressure. In 1,000 replayed incidents, the old policy safely resolves 720 (72%) but causes 35 incorrect context removals (3.5%). Its required guardrails are at least 90% safe resolution and zero canonical-context loss. Threshold tuning to 20 seconds resolves 830 (83%) and causes 18 losses; tuning to 45 seconds resolves 760 (76%) and causes 5 losses. None passes: this is a structural failure, not a threshold-selection problem. The example numbers are illustrative.

A replacement policy uses verified token pressure, checkpoint age, tenant-specific rate limits and explicit canonical-context protection. On a separate 1,000-incident holdout, it safely resolves 930 (93%) with zero canonical-context losses. This justifies *considering* a narrowly scoped rollout, not claiming proven production safety: rare catastrophic events require additional adversarial tests, human sign-off and live monitoring.

Safe retirement and replacement procedure

1. Freeze the failing version and preserve policy hashes, incident traces, scheduler decisions, human overrides and affected workflow/checkpoint IDs.
2. Protect continuity: inventory in-flight compaction jobs, drain or fence old workers, version policy decisions, and ensure retrying Celery tasks cannot revive the retired controller.
3. Test independent alternatives against historical incidents, new workload slices, adversarial cross-tenant cases and provider outages. Compare safe recovery, latency, cost, unnecessary interventions and *any* missing canonical facts.
4. Select a fallback: disable automatic compaction if evidence or safety is inadequate. Reduced throughput is preferable to silently discarding required context.
5. Shadow then canary the candidate on isolated tenants or workflow cohorts, with immutable routing assignments, explicit stop conditions and a versioned rollback plan. Shadow execution must not mutate production memory.
6. Verify after cutover: audit which policy actually handled each job, detect stale queues or conflicting controllers, and confirm recovered SLOs *and* delayed context-quality outcomes. Keep a named owner and retirement record.

Failure modes and trade-offs

• Wrong comparison: replay data resembles training incidents; hold out unseen provider failures, workloads and tenants.
• Misleading recovery metrics: faster queue draining conceals missing task requirements; verify final answers against canonical source facts.
• Rollback hazard: switching config does not cancel scheduled tasks; use generation IDs, fencing and idempotent execution.
• Shared-resource leakage: candidate and control may interfere through caches, rate limits or queue contention; report contaminated cohorts.
• Cost and availability: conservative fallback increases latency and operator toil; replacement demands more engineering and longer observation than simple recalibration.

Distinct from earlier lessons: Questions 253–254 cover generalization and drift detection; 255 tunes an existing policy; this lesson establishes criteria for structural replacement or retirement, safe fallback, and eliminating residual old-policy execution.

Further reading

• Google SRE Workbook, *Canarying Releases*: https://sre.google/workbook/canarying-releases/
• NIST AI Risk Management Framework: https://www.nist.gov/itl/ai-risk-management-framework
• LangGraph durable execution documentation: https://docs.langchain.com/oss/python/langgraph/durable-execution