What metadata should accompany a registered production model?

A production model should arrive with a traceable, reviewable metadata record—not just a model file. The record should answer: *What was built? Which data and code produced it? How was it evaluated? What inputs does it accept? Who approved it? Where is it running, and how can we recover?*

A practical example

A food-delivery company registers delivery_eta, version 7, to predict the number of minutes until an order arrives. Its metadata might record:

• Identity and purpose: delivery_eta, version 7; predicts delivery duration in minutes for the supported delivery regions.
• Provenance: Training run run_4821, source model artifact URI and immutable artifact checksum.
• Data: Training dataset snapshot orders_2026w43_v3, feature schema eta_features_v4, evaluation snapshot holdout_2026w43, and their immutable identifiers.
• Code and environment: Git commit e8ab... (full SHA in real records), dependency lockfile identity, and training image digest.
• Evaluation evidence: Holdout MAE 3.6 minutes, evaluation protocol eta_eval_v4, sample count, slice results, baseline comparison, and test report URI.
• Input/output contract: distance_km and traffic_index numeric inputs; a numeric eta_minutes output; missing-value rules and supported units.
• Ownership: Responsible team delivery-ml, operational owner/on-call contact, and model documentation link.
• Review: A recorded approval decision with reviewer identity, timestamp, policy/check version, and evidence reference.
• Deployment: Target service eta-predictor, production region, deployment revision, runtime image digest, release timestamp, rollback reference, and monitoring dashboard.

These values are illustrative. Real approvals and model deployment states must come from the systems that actually performed those actions.

Why the metadata matters: a numerical example

Suppose version 7 achieves 3.6-minute MAE on 2,000 holdout deliveries, while the currently deployed baseline achieves 4.0 minutes on the *same* holdout set.

The relative reduction in error is:

(4.0 - 3.6) / 4.0 × 100 = 10%.

Record both measured values, the dataset version, the scoring method, the evaluation date, and important segment-level results. A headline saying “10% better” without a test dataset and metric definition is not sufficient evidence for deployment.

A 10% average improvement also does not guarantee acceptable behavior for long-distance deliveries, severe weather, or smaller regions. Attach those slice-evaluation results or links to the reports.

Essential categories of production-model metadata

1. Provenance and reproducibility

• Registered model name and the specific version identifier.
• Source MLflow run ID or logged-model ID, model artifact URI, and an integrity checksum where applicable.
• Dataset snapshot IDs, feature/schema versions, training pipeline execution ID, and source-code revision.
• Package/environment or container identity, if required to rerun training or load the model.
• Creation timestamp and the training objective.

The registry need not duplicate large datasets or notebooks. Store reliable references to separately managed sources.

2. Evaluation and limitations

• Primary metric, its direction and units, held-out dataset identity, sample size, and evaluation protocol.
• Baseline comparison, important slice results, threshold acceptance criteria, and evaluation-report location.
• Reliability, latency, robustness, security/privacy checks, and known limitations relevant to the application.
• Where applicable, a model card documenting intended use, excluded uses, risks, and maintenance considerations.

Do not collapse all evidence into a boolean passed=true; retain the measurements and reports supporting the decision.

3. Model input/output contract

A model signature states what requests should look like:

Inputs:
  distance_km: float
  traffic_index: float
Output:
  eta_minutes: float

Also capture units, valid ranges, nullable fields, preprocessing assumptions, feature ordering if relevant, and representative safe test inputs. The signed interface allows serving clients to validate requests before execution.

4. Ownership and governance

• Team and accountable owner responsible for accuracy and operations.
• Approval decision, reviewer or automated policy identity, timestamp, and policy version.
• Links to model documentation, risk assessments, incidents, and change requests.
• Retention/deprecation rules and review schedule.

Model registry tags are descriptive metadata, not independently sufficient proof of an enforced approval. Store verifiable decision events in an auditable system.

5. Deployment and support

• Current deployment service, environment, region, release ID and deployed model version.
• Serving image/build digest, activation time, endpoint contract and supported traffic scope.
• Monitoring/dashboard links, alerts and an owner to contact for incidents.
• Last known good version and rollback plan.

A model may be *registered* without being *deployed*. Deployment state should be sourced from a deployment control plane and synchronized to registry metadata as appropriate. Do not assume an alias move instantly refreshes all running workers.

MLflow: attach useful metadata

MLflow supports model-version descriptions and tags, run links/provenance, and packaged model signatures. For instance, assuming version 7 is already registered:

PYTHON
from mlflow import MlflowClient

client = MlflowClient()
model_name = "delivery_eta"
model_version = "7"

client.update_model_version(
    name=model_name,
    version=model_version,
    description=(
        "Delivery-time regression model. "
        "Validation evidence: eta_eval_v4. "
        "Owner: delivery-ml."
    ),
)
client.set_model_version_tag(model_name, model_version, "owner_team", "delivery-ml")
client.set_model_version_tag(model_name, model_version, "eval_dataset", "holdout_2026w43")
client.set_model_version_tag(model_name, model_version, "eval_mae_minutes", "3.6")
client.set_model_version_tag(model_name, model_version, "approval_reference", "change-1042")

These API calls store labels and descriptions; they do not verify the evaluation, grant access, or approve a release.

Record the model signature during model logging

For a trained scikit-learn model with a representative feature DataFrame X_sample and corresponding predictions:

PYTHON
from mlflow.models import infer_signature
import mlflow.sklearn

signature = infer_signature(X_sample, trained_model.predict(X_sample))
mlflow.sklearn.log_model(
    sk_model=trained_model,
    name="eta_model",
    signature=signature,
    input_example=X_sample.iloc[:2],
)

The variables trained_model and X_sample must be defined from a real training workflow. A signature is part of the model package, not simply a registry tag; logging a new model signature may require creating a new model version when the underlying model package is immutable.

Metadata record versus evidence source

Prefer small, searchable fields in the registry, with authoritative evidence in other systems:

• Registry: model/version identity, description, source, tags, and links.
• Tracking service: training parameters, logged metrics, evaluation artifacts, and run history.
• Data catalog: durable dataset references and feature definitions.
• Review system: approval decisions, audit trail and reviewers.
• Deployment platform: actual deployed revision, service health, rollback and runtime endpoints.

This prevents manually maintained registry values from becoming an unreliable substitute for deployment or approval records.

Pre-release completeness checks

Before a production handoff, require:

1. A resolvable artifact URI and loadable model package.
2. Traceable training input, code and environment references.
3. An explicit input/output signature and sample prediction check.
4. Appropriate baseline and slice evaluation reports.
5. An accountable owner and review evidence.
6. An approved deployment target, monitoring plan and rollback reference.

Common mistakes include registering an anonymous model file, publishing metrics without evaluation conditions, storing credentials inside tags, omitting input units, confusing model approval with actual deployment, and trusting an unverified manually entered production=true tag.

In short: A production registry entry should connect identity, reproducibility, evaluation evidence, interface contract, accountability, approval and deployment. The next lesson focuses specifically on how to enforce approval gates before promotion.