How do you safely deprecate and decommission production ML model versions and serving dependencies?

Safe production model retirement is a controlled lifecycle change—not a delete command. It removes a model version from serving and future use only after verifying that no live traffic, scheduled jobs, disaster-recovery region, rollback policy, experiment, feature consumer, or evidence obligation still depends on it. Model artifacts, deployment resources, feature contracts, and audit records have different retirement schedules.

Curriculum boundary: Question 2 explained the overall MLOps lifecycle; Question 122 covered rollback *during a release*; Question 125 covered CI/CD/GitOps deployment; Question 129 covered failover/DR; Question 133 optimized serving cost; and Question 135 defined audit-ready evidence and retention. Question 136 explains the end-of-life workflow for deployed model revisions, endpoints, consumers, artifacts, secrets, monitoring, and rollback dependencies—without repeating those earlier deployment mechanics.

Beginner-friendly example: retiring late-risk r27

A delivery company has moved its checkout risk inference from model late-risk/r27 to late-risk/r28.

An engineer sees 0% apparent customer traffic on r27 and wants to delete the old weights to save storage. But:

• A nightly batch scorer still pins the r27 digest.
• A canary rollback manifest still names r27 as a known-good release.
• A secondary region has not yet warmed r28 and is configured to serve r27 after a primary-region outage.
• A shadow comparison may continue requesting r27 predictions without counting them as customer-serving traffic.
• A dispute may require reconstructing a historical decision that used r27, subject to approved evidence and retention requirements.

The correct choice is to deprecate r27, migrate dependencies, and retain approved recovery/evidence assets until the retirement gates pass. Low production traffic is one useful signal, not sufficient proof that deletion is safe.

Distinguish lifecycle states

• Active: Authorized for new deployments and/or live requests according to published policies.
• Deprecated: New use is discouraged or denied by policy; existing authorized consumers continue temporarily with an owner and removal deadline.
• Draining: No new traffic or jobs are assigned to the old revision; existing work, retries, buffered events, pinned sessions and in-flight requests finish or are safely reconciled.
• Retired from serving: No serving resources or consumers should execute the revision. Compatible rollback and reconstruction assets may still be retained.
• Retained for recovery or evidence: Model digest, minimal runtime, manifest or audited evidence remains accessible under restricted permissions for a documented period.
• Eligible for deletion: All live and recovery references are gone, no legal/privacy/contractual hold applies, and an accountable owner approves data/artifact cleanup.
• Deleted and verified: The correct assets are removed from applicable active stores and deletion records are reconciled, including any permitted backup expiry plan.

These states are a recommended operational policy, not a built-in MLflow state machine. Implement them through registry tags, release metadata, change-control records and deployment policy. MLflow version aliases are mutable references; a change to an alias does not necessarily stop processes that already loaded the old bytes. MLflow's own documentation warns that deleting registered models or versions is irrevocable and should be approached cautiously.

Seven-step decommissioning workflow

1. Inventory the model's full dependency graph. Start with immutable model_digest, image digest and feature-contract revision. Search:

• All regions, zones, clusters and serverless functions, including standby and ephemeral review environments.
• Routing configurations, stable/canary/fallback branches, shadow experiments, cached model loaders and long-lived worker processes.
• Scheduled batch jobs, queued async requests, event-stream processors, checkpoints and replay paths.
• Feature schemas, transformation code, cache namespaces, datasets, output consumers and business policies.
• CI/CD templates, GitOps manifests, registry aliases, monitoring rules, SLO dashboards, alert routes and runbooks.
• Audit/reconstruction records, legal holds, DR restore manifests, backup locations, approvals and exception registers.

A registry search alone is not enough. Compare actual execution identities to declared configuration, and require owners to confirm hidden or rarely running consumers.

2. Define the replacement and cutover contract. Identify the successor model, compatible feature schema, preprocessing, output semantics, business thresholds and intended fallback policy. Establish the minimum observation period, acceptable invalid-decision rate, latency SLO, label maturity and rollback boundary. If r28 is incompatible with r27 output classes, treat every consumer migration as a contract change.

3. Deprecate before removal. Mark r27 as deprecated through a platform-specific tag or policy, announce an explicit sunset deadline, reject *new* deployments where safe, and notify dependent teams. Do not remove the model from active jobs until those jobs are migrated or explicitly waived. Preserve a named owner and approved rollback target.

4. Stop new use and drain existing work. Update desired deployment state, route weights, scheduled-job specifications and client configuration. Wait for existing workers, queues, retries and experiment assignments to finish or safely transfer. If a model is embedded in a long-running stream operator, perform a compatible checkpoint/state migration rather than deleting the backing image during recovery.

5. Verify true zero usage. Check actual request counts per release, model loader events, job manifests, batch output lineage, stream checkpoints, queued work and client references across a sufficient window. A missing metric does not prove zero use. A rarely invoked monthly job needs a review covering its cadence, not just a 24-hour telemetry window.

6. Separate cost cleanup from evidence retention. Remove obsolete *compute* and routing dependencies after draining; retain restricted model artifacts, feature specifications, release manifests and decision-provenance evidence as required for rollback, audit or investigation. Validate applicable privacy, retention and legal-hold rules with the responsible owners. Not every record needs the same retention period, and indefinite retention can also be a privacy/security risk.

7. Delete only after explicit gates and verify. Confirm no active, standby, rollback, historical replay or regulatory dependency; apply authorized registry/object-store operations; inspect results in each region/environment; log what was removed and what remains under retention. Test that production, scheduled processing and DR rehearsals still work after the safe cleanup. Keep a final tombstone with immutable IDs, time, approvers and deletion evidence.

Numerical example 1: why a low-volume batch dependency matters

Model r27 handles 0 online requests during the last seven days. A nightly prediction job scores 50,000 records using r27, while r28 has already replaced its online endpoint.

• Seven-day online r27 count = 0.
• Seven-day batch r27 count = 7 × 50,000 = 350,000 predictions.
• If the registry-only cleanup deletes r27's artifact, the next batch job may fail all 50,000 records or fall back to an unapproved model.
• A false “unused model” conclusion arose because only synchronous inference was examined.
• Migration gate: update the nightly job to pin r28, validate the output schema and row counts, complete a representative run, reconcile consumer outputs, and check no queued r27 jobs remain.

Lesson: Retire the version only after checking online, batch, async, streaming, experimental and DR consumers independently. The correct time window depends on the least frequently scheduled legitimate consumer.

Numerical example 2: rollback versus retirement time

Suppose a team requires a 14-day rollback eligibility window after promoting r28 and an additional seven-day observation period for a slower downstream label outcome. Deployment happened on November 1.

• Earliest date the simple rollback window ends: November 15.
• Earliest date slower outcomes have satisfied the additional wait: November 22, assuming the observations started at deployment and all prerequisite data matured.
• Actual deletion may be later because monthly jobs, historical replay commitments, audit holds, or the DR site's model migration could still depend on r27.
• The dates are illustrative policies, not universal requirements. A health issue at day 12 may justify keeping r27 longer; model rollback itself may be impermissible if feature schemas have changed irreversibly.
• Always test a specific known-good replacement for the intended emergency recovery before removing old assets.

Keep an explicit distinction:

• Rollback still required: retain an executable, integrity-verified release and supporting feature compatibility.
• Rollback no longer allowed but audit reconstruction required: retain only policy-approved reconstruction artifacts/evidence; avoid leaving a dormant production endpoint open.
• No remaining approved purpose: perform authorized cleanup and verify storage/key/backup disposition.

Numerical example 3: cleanup cost, storage and recovery risk

An old inference revision uses:

• 2 idle warm workers at an illustrative $0.70 per instance-hour.
• 400 GB of model images, mirrored artifacts and test fixtures at an illustrative $0.025 per GB-month, before versioning/backup overhead.
• A restricted 10 GB evidence bundle that must remain available under the organization's approved policy.

Compute released by safely stopping the old workers: 2 × $0.70 × 24 × 30 = $1,008 / 30-day month.

Nominal model-artifact storage: 400 × $0.025 = $10 / month.

Even if deleting every old artifact saved $10/month, deleting before the rollback/DR constraints pass can compromise recovery. A sensible sequence is:

1. Drain old serving traffic.
2. Remove unnecessary live compute and network routes.
3. Preserve the minimal pinned executable rollback package while the rollback window is open.
4. Retain the 10 GB evidence bundle according to policy.
5. Delete the remaining eligible data after approval and verification.

The actual savings depend on billing, shared node utilization, storage classes, backups, egress and whether removal truly reduces paid capacity. Do not count a stopped Kubernetes Pod as a cost saving if the underlying node stays billed and cannot be scaled down.

Numerical example 4: the absence of traffic may be misleading

An older model is used by a monthly reconciliation job on the first business day. During 21 calendar days no requests appear in its endpoint metrics.

• A 7-day or 14-day request-count check returns zero.
• The monthly workload is not necessarily inactive.
• An inventory search of scheduler configuration shows an enabled job whose next invocation is still pending.
• Deletion based solely on the 21-day counter would break a legitimate dependency at the next run.

Use registry/job/control-plane inventory plus runtime telemetry, and make the observation window cover the longest approved cadence. Inventory confidence matters more than a fixed number of days.

A safe retirement policy (illustrative, not executable vendor YAML)

YAML
retirement_request:
  model: late-risk
  retiring_release: r27
  model_digest: sha256:<verified-r27-digest>
  replacement_release: r28
  owner: inference-platform
  policy:
    status: deprecated
    reject_new_deployments: true
    approved_observation_window_days: 14
    require_no_actual_usage:
      - online_primary
      - canary_and_shadow
      - approved_fallback
      - batch_and_async
      - streaming_and_replay
      - standby_and_disaster_recovery
    require_job_and_manifest_inventory: true
    require_rollback_decision: true
    require_legal_hold_check: true
    require_feature_contract_review: true
  cleanup:
    drain_before_delete: true
    keep_recovery_package_until_release: true
    preserve_governed_evidence: true
    require_independent_verification: true

This is a policy example. Implement its actual checks in your deployment controller, model registry, scheduler, feature infrastructure and access-controlled change workflow. A written checklist that can be waived silently is not an effective gate.

How to automate dependency and deletion gates

A retirement controller can build a reconciliation report from multiple sources:

• Registry model version and aliases; actual immutable content digest.
• GitOps manifests in every environment and region, including rollback configurations.
• Kubernetes/managed-serving workloads and their observed model digest.
• Queue backlog, job scheduler definitions, job run histories and delayed retries.
• Stream consumer versions, checkpoints and state replay dependencies.
• Feature-store owners and transformation/contract dependencies.
• Artifact store objects/versions, retention flags and legal-hold markers.
• Audit evidence requirements, incident holds and legal/privacy exceptions.

Only if all hard gates pass should the workflow propose a deletion change. Use two-step authorization for irreversible/high-risk operations and a dry-run report listing candidate resources and expected savings. Prefer staged, narrowly scoped deletion over broad wildcard cleanup. MLflow notes that deleting a registered model can delete all its versions; cloud model registries likewise may require endpoint undeployment before a model can be removed. Never execute an unreviewed bulk delete from a name prefix alone.

Recovery safety: When using object-store retention features such as Amazon S3 Object Lock, deletion may be blocked by retention or a legal hold. A simple delete can create a delete marker while a protected historical object version still exists. Treat “DELETE succeeded” as distinct from “protected bytes permanently removed”; verify object-version disposition and legal/backup policy before claiming erasure.

Practical CI/CD validation checks

• Replacement contract: r28's prediction schema, thresholds, feature transformation and downstream consumers match approved expectations.
• No new references: A registry/release policy rejects new r27 deployments and prevents new jobs from pinning a deprecated release without explicit exception.
• Draining: No r27 request, queue task or stream partition remains active, based on trustworthy observed evidence.
• Failover readiness: Regional standby and restore manifests resolve r28 or another explicitly approved emergency release.
• Rollback decision: Named owner confirms rollback window is closed or an alternate tested recovery path exists.
• Audit integrity: Historical r27 decisions remain explainable using retained evidence; deletion does not corrupt required provenance.
• Deletion precision: Candidate IDs and content digests exactly match the approved plan; no shared registry/model artifacts or feature datasets are selected by mistake.
• Post-cleanup proof: Production SLO, batch completeness, stream freshness, DR runbook and monitoring remain healthy, and prohibited r27 loads fail as intended.

Release-blocking checks should verify the actual deployment state rather than merely a Git pull request that plans to remove an old release.

Failure scenarios and operational runbooks

1. Cleanup breaks a nightly batch job

• Symptoms: Runtime cannot load r27; no output manifest or incomplete scoring.
• Diagnose: Check scheduler and historical job manifest, pinned model digest and deleted registry version.
• Contain: Pause unsafe output publication; restore from an approved retained package if permitted or migrate/run against a validated replacement.
• Prevent: Include scheduled-job inventory and one verified full run before retirement.

2. Disaster-recovery failover points to deleted r27

• Symptoms: Primary region works; standby cannot start its model.
• Diagnose: Compare recovery-region release manifest and local model availability.
• Contain: Do not blindly route customers into a broken region; use the last verified healthy DR release or a business-approved unavailable mode.
• Prevent: Test standby loading and traffic routing before deleting rollback artifacts.

3. “Zero requests” was caused by broken metrics

• Symptoms: Model appears unused but a stream processor still loads it.
• Diagnose: Compare actual cluster model digests, consumer manifests and model-load events with telemetry coverage.
• Contain: Halt deletion and restore observability.
• Prevent: Require multiple independent positive proofs of no use; missing metrics are unknown, not zero.

4. A feature schema used by two models is removed

• Symptoms: r27 is retired; another endpoint using traffic-features/v4 fails.
• Diagnose: Map feature contract ownership and actual consumers, not only the model record.
• Contain: Restore compatible feature pipeline where feasible, or block unsafe predictions.
• Prevent: Give feature contracts their own versioned retirement gates and consumer acknowledgments.

5. A registry alias is changed, but old Pods keep serving

• Symptoms: Alias points to r28 but observed runtime still says r27.
• Diagnose: Inspect Pod model digests, image/config and startup caching.
• Contain: Use the normal controlled rollout/drain procedure.
• Prevent: Judge actual runtime identity; aliases are pointers, not live replacement guarantees.

6. Historical evidence disappears unexpectedly

• Symptoms: A disputed r27 decision cannot be reconstructed.
• Diagnose: Compare retention policy, legal hold, model/feature manifest, audit pointer integrity and backup expiration.
• Contain: Preserve surviving evidence and escalate to the approved privacy/legal owner; do not invent deleted provenance.
• Prevent: Separate deletion of serving compute from retention of authorized decision evidence.

7. Delete operation looks successful but protected versions remain

• Symptoms: Object-storage listing looks empty, but billed protected versions persist.
• Diagnose: Inspect versioned objects, delete markers and retention/legal-hold policy.
• Contain: Stop claiming complete erasure, document the protected objects and their lawful expiry/review process.
• Prevent: Verify version-aware disposal and distinguish active namespace cleanup from permanent retention-governed deletion.

Serving-mode-specific concerns

Synchronous endpoints

• Stop new route assignment, drain active requests and sessions, check fallback/cache/experiment references, and verify health of the replacement model.

Asynchronous inference

• Respect queued jobs submitted under r27, long-running jobs and retry semantics. Choose a documented migration or completion path; never silently rescore with different weights after a retry unless the business contract permits it.

Batch scoring

• Update schedules, snapshots, output contracts and rerun procedures. Preserve old run provenance and verify exactly one complete new-version output set.

Streaming inference

• Coordinate checkpoints, keyed state and operator schema, retained event replay and downstream idempotency. Old release references may survive long after no new events arrive.

Shadow experiments and cached predictions

• Stop shadow replay and retire comparison data under its own data policy. Invalidate or expire model-version-specific cached scores without cross-tenant exposure or a sudden unsafe cache stampede.

Multi-region serving

• Verify each region's actual artifacts and last-known-good manifests. Deletion in a shared artifact store can break both live and standby environments simultaneously.

Monitoring and governance of the retirement program

• Usage: Requests and business actions per immutable model release, plus batch/stream/async usage and model loader attempts.
• Coverage: Fraction of known model consumers with current owner, release, feature contract and next-use timestamp.
• Draining: Old-version active workers, queued tasks, experiment assignments, session affinity and incomplete batch jobs.
• Recovery: Known-good rollback availability, DR regional model readiness, restore-test age and unresolved rollback dependency.
• Retention: Artifact object counts/bytes by policy class, active hold status, scheduled deletion windows, audit receipt availability and protected backup versions.
• Change control: Deprecation notice, affected-owner acknowledgments, deletion request approvers, exceptions and completion evidence.
• Business outcomes: Replacement SLO, valid-prediction fraction, delayed model-quality metrics, downstream contract failures and customer complaints.
• Economics: Realized compute and storage reduction, not just planned instance count; retained costs for mandated evidence/standby.
• Security: Unexpected loads of a deprecated model, orphaned service credentials, stale routes, insecure old images and unowned endpoints.

Engineering trade-offs and limitations

• Retain longer: Safer for rollback, slow consumers and audits, but increases attack surface and cost. Retention must remain justified and secured.
• Delete quickly: Frees resources and reduces stale-service exposure, but risks broken disaster recovery and irreversible loss of needed evidence.
• Strict block on new deployments: Prevents reintroduction, but requires a safe exception process for recovery cases until retirement closes.
• Centralized retirement automation: Consistent gates and audit, but incomplete resource inventory can still produce false safe-to-delete conclusions.
• Shared model/feature infrastructure: Reduces duplication but makes dependency mapping and deletion coordination harder.
• Artifact-only archival: Preserves weights cheaply, but may not enable execution if runtime image, preprocessor, dependency versions or compatible features are missing.
• Versioned object storage: Better recovery and audit, but delete markers and legal holds complicate erasure/cost accounting.
• Observation windows: Longer periods find rare consumers but delay cleanup; inventory of schedules and dependencies is necessary at any window length.

Common production mistakes

• Deleting old model bytes immediately after a successful canary.
• Assuming no online traffic proves no batch, async, shadow or stream consumer.
• Treating a registry alias change as proof that all serving processes loaded a new model.
• Removing old image, preprocessing or feature artifacts before the rollback/DR contract ends.
• Forgetting monthly, on-demand or rarely run jobs.
• Deleting shared feature definitions used by unrelated models.
• Ignoring queued retries, cached predictions and event replay dependencies.
• Stopping Pods but counting savings while idle nodes remain billed.
• Retaining every raw input forever “for audit” despite data-minimization duties.
• Confusing an S3 delete marker with permanent removal of retained object versions.
• Running wildcard deletion without a reviewed, digest-pinned dry run.
• Marking “retired” before verifying old-version requests and batch outputs have stopped.
• Having no owner for the last evidence/legal hold and no documented date to revisit it.
• Failing to check that replacement model quality remains acceptable for affected cohorts.

Retirement acceptance checklist

• Named owner, successor revision and explicit business/technical retirement rationale.
• Complete live and dormant consumer inventory; all relevant service modes and regions accounted for.
• Deprecation notice, blocked new references and approved exception handling.
• Zero confirmed remaining old-version work (not merely missing telemetry), with cadence-aware observation.
• Replacement and DR/rollback procedures demonstrably safe and tested.
• Retention, privacy, compliance and legal holds reviewed per artifact/evidence class.
• Dry-run deletion scope reviewed against exact immutable identifiers.
• Serving, artifact, feature and credential cleanup executed in deliberate stages.
• Independent post-deletion checks of active use, model registry, object versions, monitoring and customer SLOs.
• Audit tombstone records what was retired, what was retained, why, by whom and when.

Takeaway: Decommission a production model only after the entire system—not just the HTTP endpoint—has stopped depending on it. Drain traffic and jobs, confirm replacement and recovery paths, preserve required evidence, remove eligible resources, and verify actual deletion. Safe retirement is the final gated stage of model deployment lifecycle management.