How should experiments, runs, and run identifiers be organized?

Organize ML experiment tracking around three distinct identities: an experiment groups related work, a run represents one execution, and an immutable run ID identifies that execution unambiguously. Human-friendly names and tags make records discoverable, but they should never replace the tracker-generated run ID.

Beginner-friendly example

A delivery company is improving an estimated-delivery-time predictor. The team creates one experiment named delivery-time/eta-regression and performs three training runs:

• baseline-linear-lr001: learning rate 0.01; validation MAE 5.2 minutes.
• xgboost-depth4: maximum tree depth 4; validation MAE 4.8 minutes.
• xgboost-depth6: maximum tree depth 6; validation MAE 4.5 minutes.

All three belong to the same experiment because they investigate the same prediction task. Each receives its own unique system-generated run ID, even when a human-readable run name is repeated.

The best observed MAE is 4.5 minutes, an improvement of (5.2 - 4.5) / 5.2 × 100 ≈ 13.46% over the baseline, assuming the same evaluation dataset and metric definition.

A useful hierarchy

Project: delivery-ml
└── Experiment: delivery-time/eta-regression
    ├── Run: baseline-linear-lr001    → run_id: [unique ID A]
    ├── Run: xgboost-depth4           → run_id: [unique ID B]
    └── Run: xgboost-depth6           → run_id: [unique ID C]

The project level may be an organizational convention or a separate feature of the tracking product. In MLflow, experiments are native containers for runs; a “project” hierarchy can be expressed through experiment naming conventions and tags.

How should experiments be grouped?

Use one durable experiment for one meaningful comparison scope, such as a business problem, model family investigation, or clearly defined evaluation objective:

• delivery-time/eta-regression for ETA regressors.
• fraud/transaction-classification for fraud classifiers.
• support/ticket-routing for ticket categorization.

Keep comparable candidates together when they share the same goal and evaluation protocol. Create another experiment when the task, target, or evaluation contract changes enough that comparing runs directly would mislead.

Avoid creating one experiment per hyperparameter trial: that scatters related runs and makes comparison difficult. Also avoid putting unrelated models into one generic all-training experiment.

What distinguishes a run?

Each run is one actual attempt or execution with its own identity. It may represent:

• Training a candidate with one configuration.
• Re-evaluating a checkpoint under a specified evaluation protocol.
• A cross-validation fold or trial inside a larger sweep.

Use separate runs when the execution is meaningfully separate. For cross-validation or parameter sweeps, parent/child runs can keep the trials grouped without losing each trial's identity.

A retry is not automatically a new run. Whether retry attempts remain within one tracked run or create separate runs depends on your application and platform. Make that policy explicit, and store an attempt ID when troubleshooting retries matters. Keep a higher-level job request or pipeline execution ID for correlation.

Run ID versus run name

• Run ID: Machine-generated, unique within the tracking system, and stable for API lookups and artifact references. Treat it as the canonical identity.
• Run name: Readable label such as xgboost-depth6-seed42. It can be duplicated or edited; useful for humans, not a safe database key.
• Experiment ID: Stable system identifier for the containing experiment; do not confuse it with experiment display name.
• Pipeline execution ID: External orchestrator identity. Record it as a tag/metadata field to correlate the tracker with the pipeline.

For example, two independent runs both named baseline must not overwrite or merge, because their run IDs differ.

Practical MLflow example

PYTHON
import mlflow

mlflow.set_experiment("delivery-time/eta-regression")

with mlflow.start_run(run_name="xgboost-depth6-seed42") as run:
    canonical_run_id = run.info.run_id
    mlflow.set_tags({
        "team": "delivery-ml",
        "model_family": "xgboost",
        "dataset_version": "orders-w44-v1",
        "evaluation_protocol": "holdout-v3",
        "pipeline_execution_id": "pipeline-1042",
    })
    mlflow.log_params({"max_depth": 6, "random_seed": 42})
    mlflow.log_metric("validation_mae_minutes", 4.5)
    print("Run ID:", canonical_run_id)

This example logs an illustrative result and shows how a system-generated identifier is obtained; it does not train an XGBoost model. The experiment name is a naming convention, not a nested folder automatically created by MLflow.

Practical conventions for production teams

1. Agree on experiment naming. Use a predictable domain/task or team/use-case format; avoid dates or hyperparameters in experiment names unless they define a genuinely separate study.

2. Keep run names short and descriptive. Include only a few useful characteristics, such as model family, strategy, or seed; detailed configuration belongs in structured parameters.

3. Never construct run IDs from timestamps or usernames alone. Let the tracking service generate its canonical ID. Timestamp collisions, human edits, and concurrent executions make self-invented IDs fragile.

4. Use tags for filtering. Common tags might be team, stage, dataset_version, model_family, and evaluation_protocol. Use a controlled vocabulary so one team doesn't log production while another logs prod.

5. Correlate across systems. Store the tracker run ID in pipeline logs and model-registration metadata; store the pipeline job ID in the tracking run. A model candidate should be traceable back to the exact execution that created it.

6. Separate research from deployment status. An experiment run may produce a candidate without making that candidate production-ready. Model approval and deployment are separate controlled processes.

How should a team handle nested experiments?

Suppose a hyperparameter search launches 10 trials. One parent run can represent the search request and 10 child runs can represent individual trials. The parent helps navigation, while the children own their metrics and parameters. If you log one run per fold, explain whether the reported summary metric is an aggregate across folds or a metric for one fold.

Nested tracking structures are helpful but optional. Do not make a complicated hierarchy for a tiny study.

Common mistakes

• Treating display names as guaranteed unique identifiers.
• Making one experiment per run or putting unrelated objectives in the same experiment.
• Reusing an old run ID for a distinct training attempt.
• Logging different evaluation protocols under one metric name without tags or an explanation.
• Losing the connection between an orchestration task and the resulting experiment run.
• Embedding credentials, private customer records, or sensitive identifiers in names or tags.
• Assuming a tracking run and a production model version are the same object.

In short: Group comparable work into experiments, represent individual executions as runs, and use system-generated immutable run IDs as canonical references. Consistent names, tags, and cross-system correlation make experimentation searchable without compromising identity.