How do you manage training job outputs and artifact retention?

Training artifact management is the practice of storing every important output from an ML training run with a stable identity, traceable provenance, appropriate access permissions and an explicit retention or deletion policy. The objective is to make successful runs reproducible and recoverable without retaining every temporary file forever.

Question 108 explained how distributed checkpoints are created and restored during failures. This lesson focuses on the broader lifecycle after an artifact is produced: where it lives, how it can be found, which artifacts are protected, when storage classes change and how cleanup happens safely.

What does a training job produce?

• Final model artifacts: Serialized weights, model format, tokenizer/preprocessing components, inference signature, model environment and inference configuration.
• Intermediate checkpoints: Potential recovery states and step/epoch markers. Their *retention* can be shorter than a published model's retention.
• Evaluation artifacts: Metrics, confusion plots, holdout predictions, slice analysis, error reports and comparison evidence.
• Experiment evidence: Configurations, random seeds, code revision, runtime/container digest, training logs and resource/cost summaries.
• Data/feature references: Immutable dataset snapshot IDs, feature-view versions and split definitions. Prefer pointers/checksums over copying large datasets into each run when the underlying snapshots have an appropriate lifecycle guarantee.
• Temporary outputs: Scratch files, cache, intermediate tensors and staging uploads. These should be short-lived, not confused with durable reproducibility artifacts.

A metrics value stored in an experiment tracker is often metadata, while a model binary or plot is an artifact. MLflow documents this separation: its tracking backend stores parameters, metrics and tags; its artifact store holds potentially large files.

Practical example: organizing delivery-time experiments

A food-delivery company trains a new ETA model every day. It uses a structured object-storage path:

s3://example-ml-artifacts/
  team=delivery/
    project=eta/
      experiment=eta-weekly/
        run=run-20261106-042/
          manifest.json
          model/model.onnx
          reports/evaluation.json
          reports/error-slices.json
          config/training.json
          checkpoints/step-6000/
          logs/training-summary.json

This is an illustrative storage layout, not an actual bucket or a required MLflow path. Use storage provider object-version IDs or content hashes to distinguish immutable revisions. Make run IDs globally unique within the required scope and avoid overwriting old successful artifacts.

An immutable manifest might record:

• run_id, experiment_id, model_name and execution timestamp.
• Git commit and container image digest.
• Dataset snapshot ID, approved split/feature definition revisions and label specification.
• Training configuration, seed and dependency-lockfile reference.
• Model and checkpoint artifact URIs, sizes, checksums and creation timestamps.
• Validation metrics, evaluation report URI and artifact format.
• Parent/child trial relationship, registry model/version and deployment references.
• Retention class, owner, protection status and deletion eligibility.

The storage URI alone is a *location*, not a guarantee that content remains immutable; prefer immutable or versioned objects and checksum verification.

Why lineage matters

Suppose an incident occurs after ETA model v18 is deployed. Its registry entry points to a final model artifact, which points to training run run-042. The run manifest links to snapshot orders-20261102-a, code commit abc123 and feature definition eta-features-v4.

This chain answers:

1. Which exact model produced the prediction?
2. Which dataset and feature definitions were used to train it?
3. Which code and environment should recreate it?
4. Which tests and evaluation evidence approved it?
5. Which artifacts must not be deleted while this model is serving or available for rollback?

Maintain bidirectional references where practical: a model should locate its run, and retention tooling should be able to identify all active registered models that rely on an artifact. A registry reference to a deleted model binary is not a useful rollback strategy.

Numerical storage-cost and retention example

Assume the team runs 30 experiments each month. Each run produces:

• One 2 GiB final model.
• Five 2 GiB checkpoints.
• 0.5 GiB of reports and logs.

Total per run: 2 + (5 × 2) + 0.5 = 12.5 GiB.

If everything is retained:

• Monthly growth = 30 × 12.5 = 375 GiB.
• Twelve months of runs = 4,500 GiB, approximately 4.39 TiB.

Now define a hypothetical tiered policy:

• Retain all final models and reports for 12 months for reproducibility and review.
• Retain all intermediate checkpoints for 30 days unless a run is explicitly protected.
• Keep registered production and rollback model versions for as long as they are referenced, with a separate reviewed release-retention rule.
• Transition eligible older artifacts to colder storage if retrieval-time and minimum-duration costs are acceptable.

Ignoring calendar-month boundary effects, the approximate steady-state storage after 12 months would be:

• Final models plus reports: 30 × 12 × 2.5 = 900 GiB.
• Last month of checkpoints: 30 × 10 = 300 GiB.
• Total ≈ 1,200 GiB rather than 4,500 GiB.

Estimated reduction = 4,500 - 1,200 = 3,300 GiB, roughly 73.3% less retained storage. This simplified estimate assumes every final model and report is retained, no extra replicas, versioned objects, incomplete uploads or protected checkpoints, and no per-object pricing overhead.

At an illustrative flat $0.023 per GiB-month, storage cost would be roughly:

• Retain everything: 4,500 × $0.023 = $103.50/month.
• Tiered retention: 1,200 × $0.023 = $27.60/month.
• Illustrative steady-state difference: $75.90/month.

The rate is not a current vendor price quote. Real costs differ by provider, region, storage class, object count, retrieval charges, minimum storage duration, request operations, network transfer and replication. Do not treat a cold-tier migration as free savings.

Define retention by artifact role, not by age alone

Scratch/temp artifacts: Delete after the pipeline succeeds and any required debugging grace period ends; clean abandoned multipart uploads and incomplete staging paths.

Intermediate checkpoints: Keep a limited count and recent window for active/restartable runs; protect a checkpoint while a job is still running or recoverable.

Completed experiment diagnostics: Keep enough logs, configs and evaluation evidence to debug and compare runs under agreed internal retention.

Registered and deployed models: Preserve model binaries and the necessary compatibility metadata while production, shadow deployment or rollback depends on them.

Regulated or legal-hold artifacts: Follow approved retention and deletion requirements, including relevant legal or organizational policy; they may override default cleanup.

Training inputs: Snapshot retention must be coordinated with model reproducibility goals. A training manifest pointing to a dataset already expired by another lifecycle rule is not sufficient for exact reruns.

These durations are business policies, not universal defaults. Define minimum retention, maximum retention where applicable, legal holds, and owner-approved exceptions.

Manage outputs using MLflow

MLflow Tracking logs run metadata and artifacts and records artifact URIs. A compact pattern:

PYTHON
import json
from pathlib import Path
import mlflow

with mlflow.start_run(run_name="eta-train") as run:
    mlflow.log_params({
        "code_revision": "abc123",
        "dataset_snapshot": "orders-20261102-a",
        "training_seed": 42,
    })
    mlflow.log_metric("validation_mae_min", 4.6)

    manifest = {
        "run_id": run.info.run_id,
        "dataset_snapshot": "orders-20261102-a",
        "code_revision": "abc123",
        "artifact_role": "candidate",
    }
    mlflow.log_dict(manifest, "manifest.json")

    # Assume this file was created by the training program.
    mlflow.log_artifact("outputs/evaluation.json", artifact_path="reports")

The example requires MLflow configuration and an existing outputs/evaluation.json. In a deployed system, use a shared durable artifact store rather than a worker-local path; the proper MLflow artifact URI can be queried through its APIs. Model artifacts should be logged with a supported model-flavor API and separately evaluated and registered as appropriate.

Important deletion nuance: MLflow documentation states that deleting a run does not immediately remove all backend metadata and corresponding artifact objects, because recovery is supported. Its mlflow gc command can permanently remove eligible deleted runs and artifacts. Do not assume delete_run instantly reclaims storage. Production cleanup requires carefully coordinated metadata and object deletion.

Lifecycle policies and safe cleanup automation

Object storage providers such as Amazon S3 and Google Cloud Storage support age-based lifecycle rules for transitioning or deleting objects. However, bucket-wide age-only deletion can remove files still referenced by active models.

A safer design is:

1. Inventory artifacts: Enumerate run IDs, object versions, total bytes and lifecycle class.
2. Compute reachability: Find artifacts referenced by production, approved, active training, rollback candidates and policy holds.
3. Select eligible objects: Apply retention rules only to unprotected artifacts past their cutoff.
4. Dry-run the deletion: Produce object/version IDs, sizes, affected runs and estimated cost/recovery impact.
5. Require review for sensitive deletion: Especially for production/rollback assets or data under retention restrictions.
6. Delete in a controlled batch: Apply object version–aware delete operations; record outcomes and errors.
7. Reconcile: Verify remaining registry references, storage size and deleted metadata records.
8. Retain audit evidence: Store cleanup manifest, approver, rule version, timestamp and exception details.

Never use a mutable object prefix as the only proof that an artifact is unused. A deployed model might read from that prefix even if the experiment run is marked deleted.

S3 versioning caveat: Expiring a current object in a versioned bucket may create a delete marker while older object versions remain billable. Noncurrent-version expiration is a separate rule. Object Lock or unfinished replication can affect deletion eligibility. Google Cloud Storage retention policies and object holds also restrict lifecycle deletion. Validate actual provider semantics before enforcing cleanup.

Access controls, encryption and artifact integrity

• Apply least-privilege read/write/delete permissions per team and environment.
• Restrict production model writes and deletion more tightly than experimental outputs.
• Encrypt artifacts at rest and in transit, with keys managed under organizational policy.
• Avoid storing credentials or unnecessary personal data in logs, manifests or model metadata.
• Verify checksums/digests on critical artifact uploads and downloads.
• Separate deployment credentials from training-job credentials.
• Audit downloads, permission changes and deletion of high-value artifacts where supported.

A model file can contain unsafe serialized code (for example, untrusted pickle objects); only load trusted artifacts using the chosen framework's documented safe-loading practices. Storage access control protects both integrity and confidentiality.

Recovery and disaster-readiness checks

A retention policy is safe only if it preserves what recovery workflows need.

Test that:

• The last approved production model and its dependencies can be downloaded and loaded.
• A supported rollback version remains accessible.
• Needed checkpoint(s) for an interrupted run are within their retention window.
• The run's dataset snapshot, feature version and environment identifiers can still be resolved.
• Model integrity and associated evaluation evidence survive partial storage outages.
• Recovery from cold or archival storage meets operational RTO and RPO requirements.

A colder storage tier may reduce ongoing cost but impose retrieval delay or minimum-duration charges. Keep rapid rollback copies in appropriate storage when rapid recovery is a requirement.

Monitoring storage and artifact retention

Track:

• Stored GiB/TiB and monthly growth by project, artifact class and owner.
• Storage and retrieval spend by class and location.
• Number of orphaned artifacts (no valid references) and broken model URIs.
• Protected artifacts held longer than their nominal retention.
• Failed/incomplete uploads and abandoned staging objects.
• Cleanup dry-run estimates, deletion outcomes and audit records.
• Oldest reproducible training run and verified rollback-model availability.
• Time to retrieve a model from each storage tier.

Review deletion failures as carefully as upload failures: retention errors may cause unexpectedly high cost, while overly broad cleanup can destroy reproducibility or production recovery options.

Trade-offs and mistakes to avoid

• Keep everything: Simplest operationally, but causes cost growth and possibly excessive data retention.
• Delete aggressively: Saves storage but risks breaking rollback, audit or model reproducibility.
• Cold-tier all artifacts: Lowers storage rates but can make urgent restore slow or expensive.
• Deduplicate identical blobs: Can save space if artifacts are content-addressed, but reference counting and garbage collection become essential.
• Version every overwrite: Helps auditability but may leave costly noncurrent versions unless the lifecycle is designed.
• Retain only the final model: Saves space but often loses evidence needed to explain how it was trained and approved.
• Rely on object age only: Can delete old yet actively deployed model versions.
• Assume metadata deletion removes files: Creates hidden storage growth in some tracking systems.
• Store everything on local worker disks: Artifacts can vanish when a container or compute node terminates.

In short: Give each training output a stable identity, durable location, ownership, lineage and explicit retention class. Protect artifacts referenced by active models, automate reviewed cleanup for everything else, and regularly test that required recovery and reproducibility paths still work.