MLOps
MLOps applies software engineering and operations discipline to machine learning. It covers reproducible training, model registries, deployment, monitoring, governance, and incident response.
How should you choose an orchestrator for an ML platform?
Choose an ML orchestrator by matching the workflow's needs to the team's infrastructure and operating capacity—not by choosing the tool with the longest feature list. The best option is the one your team can reliably deploy, debug, secure, and maintain. Questions 68–70 introduced Airflow, Kubeflow Pipelines, Prefect, and Dagster individually. This lesson gives a repeatable decision process for evaluating them, without repeating their introductions. Start with the workload, not the product Write down the expected workflow: Then answer: 1. Trigger: Nightly schedule, dataset arrival event, manual experiment, or mixed? 2. Execution: Mostly SQL and API calls, Python functions, or containerized CPU/GPU training? 3. Artifacts: Must the platform track dataset to model lineage, versions, and freshness? 4. Scale: How many workflows, runs, concurrent tasks, and accelerator jobs? 5. Infrastructure: Does the team already operate Kubernetes, Airflow, or a managed data platform? 6. Reliability: What retry, backfill, deployment approval, security, and audit requirements apply? 7. Ownership: Who will maintain workers, schedulers, permissions, logs, upgrades, and incident response? These requirements determine which candidate is worth testing. A beginner friendly selection example A food delivery company retrains one model nightly. It loads records from a warehouse, creates features in Spark, starts a managed training job, evaluates the model, and registers a candidate. Assume the work consists of: Warehouse extraction: 4 minutes Spark feature preparation: 12 minutes Managed model training: 25 minutes Evaluation and candidate registration: 5 minutes The workflow takes about 46 minutes of sequential processing , excluding queues and setup. Changing orchestrators will not magically shorten the 25 minute training job. The primary decision concerns integration, operation, monitoring, correctness, and cost . The company already uses Airflow for warehouse pipelines and has no Kubernetes platform team. Airflow is the practical initial shortlist leader because it can coordinate these existing systems without building a new cluster. If the company instead had dozens of componentized ML jobs, a well managed Kubernetes cluster, and a strong requirement to track ML artifacts, Kubeflow Pipelines could become the stronger candidate . For a smaller Python heavy team that prefers flexible flow functions, Prefect deserves evaluation . For a data platform where the central concern is the state, lineage, and quality of hundreds of feature and model assets, Dagster deserves evaluation . These are fit hypotheses , not universal rankings. Evaluate candidates using clear criteria A. Integration fit (most important in many existing platforms) Can the orchestrator start and monitor the actual warehouse, Spark, GPU or managed training, and model registry operations used by your team? Prefer documented integrations or reliable APIs over elaborate custom adapters. B. Execution and infrastructure Airflow: Good for coordinating bounded, cross system batch workflows. Kubeflow Pipelines: Strong fit for Kubernetes backed containerized ML components and typed artifacts. Prefect: Useful where Python flows and flexible execution infrastructure are central. Dagster: Useful where software defined datasets, assets, partitions, checks, and lineage are central. All four can coordinate ML work; these are differences of emphasis, not rigid restrictions. C. Developer experience and testing Ask engineers to implement the same workflow in the finalists. Assess local development, typed inputs, deployment procedures, unit tests, secret handling, and how easily they troubleshoot a failed task. D. Reliability and recovery Test a worker crash, missing input artifact, duplicate event, and partially completed external write. Check whether operators can inspect attempts, rerun affected tasks, and protect model releases. No orchestration tool removes the need for idempotent task design. E. Artifact and governance needs Do you require built in or integrated tracking from dataset snapshot to features to model evaluation? Verify what the orchestrator records natively, what needs additional services, and whether audit retention meets policy. F. Total cost of ownership Include compute and storage, paid control plane costs if applicable, the team's maintenance time, security and upgrade work, and training. An open source license does not make self hosting free. A simple weighted scoring exercise Suppose the team assigns weights totaling 100 points : Integration fit: 30 Developer productivity: 20 Reliability and operations: 20 Artifact lineage and governance: 15 Infrastructure and cost fit: 15 Rate each candidate from 1 (poor fit) to 5 (excellent fit) using hands on evidence. Compute: weighted score = sum(weight × rating / 5) For illustration, suppose Airflow receives ratings 5, 4, 4, 3, 5 . Its score is: 30×5/5 + 20×4/5 + 20×4/5 + 15×3/5 + 15×5/5 = 86/100. Suppose KFP receives 3, 4, 3, 5, 2 : 18 + 16 + 12 + 15 + 6 = 67/100. These are illustrative team assumptions, not measured product ratings . Another organization may weight Kubernetes native training and lineage more heavily and reach the opposite conclusion. Run a short proof of concept Before committing to a platform: 1. Implement the same representative training pipeline in the two strongest candidates. 2. Use a versioned input dataset and durable output artifacts. 3. Test one normal run and one failed task. 4. Attempt a safe rerun without duplicate model registration. 5. Trigger overlapping runs and verify output isolation. 6. Inspect run history, logs, metrics, and dataset to model lineage. 7. Record actual engineering effort, infrastructure requirements, runtime overhead, and operational surprises. Define acceptance thresholds before testing, such as recovery without duplicate releases, appropriate access controls, and an operator being able to diagnose a failed run from available evidence. Avoid common selection mistakes Choosing by popularity: Familiarity helps, but it cannot replace workload fit. Assuming GPU scheduling equals GPU training: The orchestrator coordinates work; the training framework and compute resources execute it. Building a large cluster for a tiny workflow: Infrastructure complexity can exceed the benefit. Equating orchestration with full MLOps: Experiment tracking, model monitoring, validation, feature management, and production release controls remain important. Ignoring migration cost: Reusing a reliable existing orchestrator may be better than adopting a new one for a marginal feature advantage. Optimizing only pipeline runtime: The greatest business benefit may come from fewer incidents, better lineage, and faster recovery rather than faster task execution. Practical recommendation Start by documenting the workload and giving your team specific requirements explicit weights. Narrow to two tools , build the same proof of concept, test failure and artifact handling, and choose using observed tradeoffs. In short: Airflow, Kubeflow Pipelines, Prefect, and Dagster can all orchestrate ML workflows. Choose on integration fit, execution model, artifact needs, team expertise, reliability, and total operating cost rather than treating any one framework as the universal winner.
How does DVC connect dataset artifacts to Git revisions?
DVC (Data Version Control) connects large dataset artifacts to Git revisions by storing small dataset tracking files in Git rather than committing the dataset bytes themselves. When a teammate checks out a Git commit, the DVC metadata in that revision identifies the corresponding dataset content. This is the beginning of the curriculum's data and artifact versioning section, after the orchestrator selection lessons (Questions 68–71). Here the focus is how a Git commit identifies a dataset . The next lesson explains how DVC caches and remote storage hold and transfer the actual bytes. Beginner friendly example A food delivery company trains a model using a CSV of completed orders. Monday: data/orders.csv contains 10,000 rows . The team tracks it with DVC and commits the tracking metadata to Git as commit A1. Tuesday: The dataset grows to 12,000 rows . After updating DVC metadata, the team commits the change as Git commit B2. Later: An engineer asks, “Which data was used for the model built from commit A1?” The engineer checks out Git revision A1. Git restores the DVC pointer that identifies Monday's dataset. DVC can then restore the corresponding bytes to the workspace if that version remains available in the local cache or configured storage . The Git revision acts as the link between code/configuration and data identity . The important point is that Git stores the small reference, while DVC manages the large artifact . A simple walkthrough From an existing Git repository: The dvc add command creates data/orders.csv.dvc and configures the actual CSV to be ignored by Git. The tracking file contains a content identity such as a hash (the exact metadata depends on the file and DVC version). A simplified illustration of a tracking file is: The hash above is illustrative, not a usable checksum . DVC generates the actual content identifier automatically. Do not edit it by hand to make a dataset appear to match. After the dataset changes: The Git commits contain different DVC metadata corresponding to different versions of the dataset. How do you recover the earlier dataset? To select an earlier project revision: The first command selects the project's code and its tracked DVC metadata at the historical Git revision. The second asks DVC to make the workspace dataset match that metadata using locally available cached data . If the needed dataset content is not local, retrieve it from configured DVC storage first, for example with dvc pull, then perform or confirm the checkout. Git checkout by itself does not download the large dataset . Before changing revisions, commit or otherwise protect any uncommitted code and data changes so a historical checkout cannot disrupt work in progress. What exactly is linked to a Git commit? Code and small configuration files: Usually committed directly to Git. DVC tracking metadata: A small .dvc file committed to Git identifies the expected dataset version. Actual dataset bytes: Stored and managed outside normal Git history, with DVC managed local or configured remote storage. Reproducible run context: The Git commit can identify code and DVC metadata together. To reproduce a specific training run, also record effective hyperparameters, environment or image version, random seed, and other relevant inputs. A Git commit is not proof that a dataset can still be recovered. If the corresponding bytes were not retained or are inaccessible, the pointer still exists but data restoration may fail. Why is this useful for MLOps? 1. Traceability: Connect a model experiment to the project revision and dataset version used for training. 2. Collaboration: Review dataset version changes through ordinary Git branches, commits, and pull requests without storing multi gigabyte files in Git. 3. Rollback: Identify and restore a previous dataset when investigating a regression. 4. Reproducibility: Repeat training with the same code and data versions, subject to other runtime and environment dependencies. 5. Reduced repository bloat: Git stores compact metadata instead of full binary dataset histories. Common mistakes Committing only code: The DVC pointer was changed but never committed, so the Git revision does not identify the intended data version. Assuming git clone includes datasets: It typically includes the DVC metadata but not the large dataset content. Using a mutable latest file: The training job silently consumes changed data without an explicit dataset identity. Forgetting to retain artifact bytes: The Git history remains but older data may no longer be recoverable. Assuming the dataset alone guarantees reproducibility: Python dependencies, preprocessing logic, environment, randomness, and model configuration also matter. When should you use this pattern? DVC is helpful when ML projects have datasets or model files too large or frequently changing to store directly in Git, but the team still wants a Git centered review and versioning workflow . For very large lake scale datasets or different governance needs, evaluate data platform native versioning approaches as well. DVC is a tool for linking data artifacts to project versions , not a replacement for every data warehouse or storage system. In short: Git records the revision; DVC metadata identifies the dataset content for that revision. Together they connect code and data versions without committing large datasets to Git. Question 73 explains where the actual data bytes are cached and shared.
How do DVC remotes and caches work for large ML datasets?
DVC uses a local content addressed cache to manage dataset bytes and a configured remote to share those bytes with teammates or other machines. A remote can be an S3 bucket, Azure Blob Storage, Google Cloud Storage, SSH server, network drive, or another supported storage location. Question 72 explained how small DVC tracking files in Git identify dataset versions. This question explains where the actual large files live and how they move between machines. Beginner friendly example A food delivery company has a 5 GB dataset of order histories. Alice and Bob both work on the same ML project. Alice adds the dataset with DVC; its content is recorded in her project's local DVC cache. Alice pushes the cached dataset to the team's shared DVC remote. Bob clones the Git repository. It contains code and DVC metadata, not the complete 5 GB dataset. Bob runs dvc pull. DVC downloads the needed content from the shared remote into his local cache and makes it available in his workspace. Now both developers can work from the same tracked dataset version without committing 5 GB of data to Git. What is a content addressed cache? DVC identifies cached content using content derived hashes rather than relying solely on a human readable file name. For illustration: The exact cache layout, hashing scheme, and representation of directory outputs depend on the DVC version and tracked target. The core idea is stable: identical content can be recognized by its content identity . If the 5 GB file changes, DVC can track the changed version under a different identity. If content is unchanged, the same cached object can be reused rather than making unnecessary new copies. For a tracked directory, DVC also maintains directory metadata referring to its constituent files. It does not necessarily treat every directory as one monolithic blob. Practical commands: configure, push, pull The following is an illustrative project setup. The bucket name is a placeholder and must be replaced with accessible storage. The d flag designates teamstore as the default DVC remote. dvc push uploads tracked data objects required by the current project state that are not already present in the destination. On Bob's machine: dvc pull obtains data matching the checked out DVC metadata, populates the local cache, and checks out the corresponding workspace files. Use dvc fetch if you only want to retrieve data into the cache without checking it out into the workspace. Important: Git's push and pull exchange Git objects; DVC's push and pull exchange DVC tracked data. They are separate operations. A Git commit does not automatically upload dataset bytes to DVC storage. Why does the cache save space? Imagine the team versions two datasets: Version A: a 5 GB directory with 100 image files. Version B: the same directory, but 5 images have changed . When DVC tracks a directory, unchanged file objects can be reused in its content addressed cache. Conceptually, if the images are equal sized and the changed ones average 50 MB , only about 250 MB of new image content may need to be added, plus small metadata, rather than storing a completely separate 5 GB directory. This is an illustrative estimate . Actual transfer and storage savings depend on file boundaries and content, the storage backend, compression if any, and DVC behavior. Editing a single large monolithic file may require storing another complete object; DVC is not guaranteed to compute binary block level deltas within that file . DVC can also link cached data into the workspace (for example, using reflinks or hardlinks when supported), reducing local duplication. The type of link and copy behavior depend on file system support and DVC configuration. How do DVC remotes help collaboration? Central sharing: Team members and CI workers can obtain matching datasets from an approved storage location. Dataset restoration: Historical data versions remain retrievable if their corresponding remote objects have been retained. Storage choice: Use cloud object stores, SSH, or approved on premises storage without hosting datasets inside Git. Less repeated copying: Cache reuse and content checks reduce unnecessary retransfers. Access controls: Storage credentials and bucket permissions can limit who reads or writes data. A remote is not a replacement for a backup strategy . Removing referenced objects, deleting a bucket, or misconfiguring retention can still make old versions unavailable even when Git metadata survives. Security and operational considerations 1. Never commit secrets. Keep credentials in environment or workload identity, or DVC's Git ignored local configuration. Do not put access keys in .dvc/config. 2. Verify access. Remote read permissions are required for pulls; write permissions are required for pushes. 3. Watch storage costs. Large dataset versions consume remote capacity; cloud downloads may create network/egress charges. 4. Handle cache growth. Local caches can accumulate unused historical objects. Review references and retention requirements before cache cleanup or garbage collection. 5. Avoid unsafe concurrent modification. Dataset files visible in the workspace may be linked to cached objects; follow DVC's guidance for your cache link type before in place editing. 6. Choose the right scale. For huge lakehouse scale datasets or millions of tiny objects, evaluate storage native versioning or directory oriented tracking rather than assuming DVC is optimal for every workload. What if the dataset is not found? If dvc pull fails, check: Is the correct Git revision checked out? Was dvc push performed for that dataset version? Is the right default remote configured? Are permissions and cloud credentials valid? Have referenced remote objects been deleted or expired? A Git pointer alone is insufficient; the corresponding cached content must still exist somewhere accessible . In short: DVC's local cache identifies and reuses the real dataset content; a DVC remote shares those cached objects between machines. dvc push uploads tracked data and dvc pull restores it. Together, caches and remotes support efficient, reproducible work with datasets much larger than a practical Git repository.
How can dataset snapshots be made reproducible across training runs?
A reproducible dataset snapshot is an immutable, uniquely identifiable copy or logical version of the exact data used by a training run. When the run is repeated, it must resolve to the same records, values, and schema—even if the source database has changed. Questions 72 and 73 introduced DVC's Git linkage and shared storage. Here the emphasis is snapshot design independent of the versioning tool : immutability, manifest files, checksums, data splits, and exact run references. Beginner friendly example A food delivery team trains a delivery time model every week. On Monday, the approved dataset contains 100,000 orders . The team publishes snapshot orders 2026 w44 v1 . On Wednesday, 5,000 more orders arrive in the source database. A developer reruns Monday's experiment on Friday. Incorrect approach: Query SELECT FROM orders again. The input now contains 105,000 rows, so this is not the original experiment . Correct approach: Read immutable snapshot orders 2026 w44 v1 with exactly 100,000 approved rows. Its version identifier and integrity metadata match the original run. The extra 5,000 rows belong in a new snapshot, not the old one. Reusing the same dataset does not guarantee identical model weights: preprocessing code, random seeds, training environment, accelerator behavior, and configuration also affect reproducibility. Six practical requirements 1. Freeze the input at a well defined boundary. Use a committed table version, a transactionally consistent database export, or files written to an isolated snapshot directory. Do not read multiple changing tables at unrelated times and assume they form one coherent snapshot. 2. Assign an immutable version identifier. Use a unique snapshot ID or storage version, such as orders 2026 w44 v1, not a mutable path named latest. A timestamp alone is useful context but not proof of content identity. 3. Publish a manifest. Record the dataset ID, creation time, data range and cutoff, schema version, row count, file list, file sizes, and relevant content digests. For a multi file dataset, record checksums per file and optionally a deterministic aggregate manifest hash. 4. Verify integrity on read. Compute and compare a cryptographic checksum such as SHA 256 over the actual saved bytes. A matching digest strongly indicates the stored bytes have not changed. If multiple files are used, check all listed files and their identities. A checksum is an integrity check, not a substitute for quality validation. 5. Preserve training/validation/test membership. Record split membership explicitly or use a deterministic split method with fixed input ordering, seed, group/time boundaries and code version. A seed alone may not preserve splits if the row order or dataset changes. 6. Capture the effective run inputs. Save snapshot ID, source cutoff, preprocessing version, feature definitions, parameters, training/evaluation split identities, container or environment version, and model artifact reference with every experiment run. Simple snapshot manifest The digest above is a placeholder. In production it must be populated from the saved file, never invented. Calculate and verify a checksum in Python Here manifest must be loaded from a trusted, previously published manifest containing the real digest. This code checks saved bytes, not whether the schema, labels, or records are meaningful. Numeric illustration: why a snapshot matters Suppose Monday's snapshot has 100,000 rows. An 80/10/10 split yields: 80,000 training rows 10,000 validation rows 10,000 test rows If Friday's query unexpectedly includes 105,000 records, an 80/10/10 split yields 84,000 / 10,500 / 10,500 , and potentially different individual rows in each group. Even if you keep the same random seed, these are not the same training and evaluation datasets. Explicit snapshot and split identities remove the ambiguity. For time series prediction, the snapshot and split policy should also prevent future labels or other information from leaking into earlier training periods. Where can snapshots live? Versioned object storage: Write immutable files under a versioned prefix; retain each referenced object version and use a manifest to capture the exact set of files. Versioned lakehouse tables: Delta Lake and similar table formats offer table versions or snapshots that let readers query a historical state. Retention and vacuum/cleanup policies must preserve versions required for reproducibility. Transactional database exports: Export records at a consistent cutoff or database snapshot, then retain the materialized result and metadata. Dataset versioning platforms: Tools such as DVC can retain identifiable datasets, but only if the relevant content remains available. An object store folder name does not automatically make the files immutable. Enforce access controls, write once or retention policies, and safe publication rules where required. How do you publish safely? 1. Create the output in an isolated staging location. 2. Finish writing all files. 3. Validate schema, counts, null rates, and business rules. 4. Compute real file checksums and prepare a manifest. 5. Publish the snapshot reference or manifest as the final committed step. 6. Prevent subsequent mutation and enforce retention for the required period. 7. Record the snapshot ID in each training and evaluation run. Readers should only consume a committed snapshot, never a partially written directory. What tools can record the identity? MLflow supports logging dataset source, digest, schema and profile alongside a run. Be careful: not every MLflow dataset digest is a cryptographic hash of all raw file bytes ; some dataset types use metadata or sampling. For a strict byte for byte guarantee, separately maintain actual file checksums or an authoritative storage version. Delta Lake's table time travel can select a historical version, provided the old transaction log and underlying files are still retained. Common mistakes Using latest as the only dataset reference. Keeping a snapshot ID but allowing its files to be overwritten. Saving a CSV while failing to preserve its source cutoff, schema, or splits. Assuming that checking the row count proves that every value is identical. Deleting old table versions or files before training run retention expires. Recording a dataset digest without knowing what exactly the digest covers. In short: Make datasets reproducible by committing immutable snapshots with unique IDs, real integrity hashes, consistent data cutoffs, explicit splits, and retained storage. Every training run should record exactly which snapshot it used. The next question connects these snapshots to the full source to feature to model lineage.
How do you record source-to-feature-to-model data lineage?
Source to feature to model data lineage is a record of how a particular model was produced: which raw datasets supplied its inputs, which processing jobs transformed those datasets, which feature versions were consumed, and which training execution created the model. Unlike simply storing a dataset snapshot ID (Question 74), lineage preserves the relationships among artifacts and transformations . It lets a team answer both “Where did this model come from?” and “What models are affected by this bad source dataset?” Beginner friendly example: a food delivery model A company predicts delivery time using order distances, traffic conditions, and courier history. Its production model has this lineage: Suppose delivery features v9 was created from 100,000 orders and courier features v4 contains one row per courier in the training cohort. The final training execution uses these two feature artifacts to produce model candidate v12. The lineage says which exact versions were used, not merely that the model uses “orders” and “courier data.” What records make this possible? Track three entity types: Artifacts or datasets: Stable dataset/feature/model identity, immutable version or digest, URI, schema reference, and creation timestamp. Executions or jobs: Pipeline run ID, task ID, code/image version, effective parameters, start/end times, and outcome. Relationships: Which artifacts an execution consumed and which it produced . A useful record is: In a real application, store lineage records in durable metadata storage and ensure that every input and output identifier resolves to an actual artifact. The JSON is an illustrative internal record, not a claim that every lineage product uses this exact API format. A tiny lineage graph you can reason about Think of lineage as directed links: 1. orders snapshot v17 was consumed by feature job 804. 2. feature job 804 produced delivery features v9. 3. delivery features v9 was consumed by train run 915. 4. train run 915 produced model candidate v12. 5. evaluation run 920 consumed model candidate v12 and produced evaluation results. These relationships are more precise than saving a flat list of dataset names beside a model. How do you capture lineage automatically? At each pipeline step: 1. Identify actual inputs: Resolve the immutable snapshot and feature artifact versions read by this execution. 2. Record execution identity: Use a stable run id and component/task ID for correlation. 3. Publish outputs with identifiers: Give each feature or model artifact a unique version or digest. 4. Emit consumption/production relationships: Connect the execution to exactly the artifacts it used and created. 5. Store transformation provenance: Record relevant code version, parameters, and schema/quality check results. 6. Confirm completeness: Verify the model has a path back to its source datasets before allowing a production release. For multi source jobs, capture which specific source feeds which output . Blindly connecting every job input to every job output can create false dependencies. Example: MLflow tracking and lineage metadata MLflow can record the source and identity of datasets used by an experiment run: This is a partial illustration: train df must already be a pandas DataFrame loaded from the identified dataset. The code logs an MLflow dataset and useful upstream identities; it does not automatically construct a complete source to model lineage graph, and the model training/logging operation must be implemented separately. Common standards and tools OpenLineage: Describes jobs, runs, input/output datasets, and extensible metadata so lineage can be collected across processing tools. Kubeflow Pipelines ML Metadata: Records artifacts and executions, with lineage graphs showing how artifacts flow through pipeline runs. MLflow: Connects experiment runs with input dataset sources and digests, parameters, metrics, and model artifacts. Data catalogs and feature platforms: Can add ownership, schema descriptions, lineage views, and feature definitions. None of these automatically supplies every relationship across all systems. Integrations, naming conventions, and metadata completeness are part of an MLOps platform's design. Impact analysis: what happens when data is wrong? Suppose engineers discover that the distance unit in orders snapshot v17 was incorrectly interpreted. Without lineage, they may search logs and guess which models used that dataset. With lineage: 1. Start at artifact orders snapshot v17. 2. Follow downstream relationships to feature job 804. 3. Find delivery features v9. 4. Continue to train run 915, then model candidate v12. 5. Check which evaluation results, registry versions, and deployments consumed that model. 6. Mark impacted candidates for investigation, regenerate correct features, and retrain where necessary. Numerical illustration: If lineage shows that 3 of 12 registered candidates descend from the affected feature artifact, engineers can prioritize those three for inspection. This is a metadata graph result, not evidence that the remaining nine are safe unless all relevant paths were captured correctly. The reverse query is equally valuable: starting from a deployed model, trace upstream to every source snapshot and processing execution. Common pitfalls Recording only dataset names without immutable versions. Losing the connection between a feature transformation's inputs and output. Attaching metadata to a task name but not to the actual run . Missing lineage for manual jobs or external processing services. Claiming a model is reproducible because a lineage graph exists: the underlying data and code must still be retained. Treating a copied or renamed artifact as a new, unrelated dataset without preserving its upstream relationships. Logging private row level customer data unnecessarily when artifact identifiers and aggregate metadata would suffice. In short: Build lineage as a graph of source artifacts → processing executions → feature artifacts → training execution → model artifact . Record exact identities and relationships at every step so you can explain a model's provenance and assess which outputs are affected when upstream data changes or fails.
What should an ML experiment tracking server store?
An ML experiment tracking server should preserve enough information to compare training attempts, reproduce their conditions, find their outputs, and audit what happened. It acts as a shared record of experiments and runs; it is not necessarily the place that physically stores large datasets or model weights. What should it record? Experiment identity: Name, purpose, project/team, and optional description so related attempts are discoverable. Run identity and lifecycle: Unique run ID, experiment association, start/end timestamps, execution status, owner or service identity, and failure reason where available. Configuration: Model type, hyperparameters, training arguments, feature selection settings, and the effective configuration used—not only defaults. Performance metrics: Validation/test metrics, training loss history when helpful, evaluation dataset identity, metric units, and the optimization direction (for example, lower MAE is better). Reproducibility references: Git commit, container or environment version, random seed, dataset/feature version IDs, relevant pipeline run ID, and dependencies. Artifact references: Where to find trained model files, plots, reports, confusion matrices, evaluation outputs, and other run artifacts, with access controls and retention rules. Searchable tags: Use case, stage, team, candidate/baseline designation, and meaningful labels that make comparisons easier. Model/evaluation context: Framework, model signature or input/output schema where supported, and the criteria used to approve or reject a candidate. The server should retain a run level record of these values . Detailed source to feature to model relationship graphs are covered in Question 75; tracking the dataset and upstream pipeline IDs here helps connect the systems without duplicating the entire lineage topic. Beginner friendly numerical example A food delivery team trains a model that predicts delivery time. Two runs use the same approved dataset: Run A: Learning rate 0.01 , 50 training epochs, validation MAE 5.2 minutes . Run B: Learning rate 0.001 , 75 training epochs, validation MAE 4.6 minutes . The tracking server stores both sets of parameters and metrics, along with the dataset version, code commit, execution timestamps, and model artifact references. Because MAE is lower is better , Run B improves validation MAE by 0.6 minutes , or about 11.5% relative to Run A : (5.2 4.6) / 5.2 × 100. That comparison is meaningful only if the evaluation protocol is compatible. If Run B was evaluated on an easier data split, the raw metric difference does not prove that its model is better. Metadata store versus artifact store A production ready tracking architecture commonly separates two concerns: The exact artifact upload path depends on how the tracking server is configured: clients may upload through the server or interact with permitted storage directly. Why split them? A SQL database is useful for searching and comparing metrics, while object storage is better suited for multi megabyte or multi gigabyte files. The backend should store enough information to retrieve and verify the relevant artifacts. An artifact location is only useful if the files are retained and the right users can access them. A tracking entry that points to a deleted model is incomplete in practice. A small MLflow example This assumes an accessible MLflow tracking server is running. The code shows logging , not a complete training script. The validation score is illustrative, not computed by this snippet. Do not send sensitive credentials or personally identifying training records as free text tags. What should a team be able to answer? Which model had the lowest validation error on the same evaluation dataset ? Which code commit, dataset version, and hyperparameters produced that run? Is the execution complete, failed, or still in progress? Where are its model weights and evaluation reports? Who ran it, when, and under what environment? Can a teammate inspect and compare the result without requesting the original researcher's notebook? Can the organization enforce access and retention rules? These are stronger requirements than simply showing a leaderboard of metric numbers. Reliability, security, and operations Authentication and authorization: Restrict who may view projects, log runs, read artifacts, or change registrations. Protect secrets via approved identity and credential management. Durability: Back up the metadata database and artifact storage. Ensure run references do not disappear during cleanup. Data minimization: Store dataset identifiers and aggregate metrics rather than uploading raw private customer data by default. Consistency: Decide what happens if metric logging succeeds but artifact upload fails; make incomplete runs visible. Governance: Define naming conventions, tags, retention periods, experiment ownership, and handling of deleted runs. Scalability: Plan for many concurrent runs and high frequency metric logging. Record useful step level curves without overwhelming the backend with unnecessary events. What should not live primarily in the tracking database? Entire training datasets and enormous model weight files as ordinary SQL columns. Raw secrets, cloud credentials, production tokens, or customer level personal records. Every transient debugging message when a log platform is more appropriate. An assumption that the tracking server alone is a feature store, a data lineage catalog, model serving platform, or production monitoring solution. Tracking servers may integrate with registries, deployment platforms, and observability tools, but these are distinct responsibilities. Limitations Experiment records document what was logged, not necessarily everything that happened. Missing dataset versions, incorrect metric definitions, mutable artifact paths, or unrecorded environment dependencies can make a run difficult to reproduce even when a record exists. The next curriculum question covers how experiments, runs, and run identifiers should be organized . Here the essential idea is what fields and artifacts must be preserved . In short: Store searchable experiment and run metadata, effective parameters, evaluation metrics, code and data references, execution state, and durable artifact locations. Keep large artifacts in appropriate storage, and protect both metadata and files with retention and access controls.
How should experiments, runs, and run identifiers be organized?
Organize ML experiment tracking around three distinct identities: an experiment groups related work, a run represents one execution, and an immutable run ID identifies that execution unambiguously. Human friendly names and tags make records discoverable, but they should never replace the tracker generated run ID. Beginner friendly example A delivery company is improving an estimated delivery time predictor. The team creates one experiment named delivery time/eta regression and performs three training runs: baseline linear lr001: learning rate 0.01; validation MAE 5.2 minutes. xgboost depth4: maximum tree depth 4; validation MAE 4.8 minutes. xgboost depth6: maximum tree depth 6; validation MAE 4.5 minutes. All three belong to the same experiment because they investigate the same prediction task. Each receives its own unique system generated run ID , even when a human readable run name is repeated. The best observed MAE is 4.5 minutes, an improvement of (5.2 4.5) / 5.2 × 100 ≈ 13.46% over the baseline, assuming the same evaluation dataset and metric definition . A useful hierarchy The project level may be an organizational convention or a separate feature of the tracking product. In MLflow, experiments are native containers for runs; a “project” hierarchy can be expressed through experiment naming conventions and tags. How should experiments be grouped? Use one durable experiment for one meaningful comparison scope , such as a business problem, model family investigation, or clearly defined evaluation objective: delivery time/eta regression for ETA regressors. fraud/transaction classification for fraud classifiers. support/ticket routing for ticket categorization. Keep comparable candidates together when they share the same goal and evaluation protocol. Create another experiment when the task, target, or evaluation contract changes enough that comparing runs directly would mislead. Avoid creating one experiment per hyperparameter trial : that scatters related runs and makes comparison difficult. Also avoid putting unrelated models into one generic all training experiment. What distinguishes a run? Each run is one actual attempt or execution with its own identity. It may represent: Training a candidate with one configuration. Re evaluating a checkpoint under a specified evaluation protocol. A cross validation fold or trial inside a larger sweep. Use separate runs when the execution is meaningfully separate. For cross validation or parameter sweeps, parent/child runs can keep the trials grouped without losing each trial's identity. A retry is not automatically a new run. Whether retry attempts remain within one tracked run or create separate runs depends on your application and platform. Make that policy explicit, and store an attempt ID when troubleshooting retries matters. Keep a higher level job request or pipeline execution ID for correlation. Run ID versus run name Run ID: Machine generated, unique within the tracking system, and stable for API lookups and artifact references. Treat it as the canonical identity. Run name: Readable label such as xgboost depth6 seed42. It can be duplicated or edited; useful for humans, not a safe database key. Experiment ID: Stable system identifier for the containing experiment; do not confuse it with experiment display name. Pipeline execution ID: External orchestrator identity. Record it as a tag/metadata field to correlate the tracker with the pipeline. For example, two independent runs both named baseline must not overwrite or merge, because their run IDs differ. Practical MLflow example This example logs an illustrative result and shows how a system generated identifier is obtained; it does not train an XGBoost model. The experiment name is a naming convention, not a nested folder automatically created by MLflow. Practical conventions for production teams 1. Agree on experiment naming. Use a predictable domain/task or team/use case format; avoid dates or hyperparameters in experiment names unless they define a genuinely separate study. 2. Keep run names short and descriptive. Include only a few useful characteristics, such as model family, strategy, or seed; detailed configuration belongs in structured parameters. 3. Never construct run IDs from timestamps or usernames alone. Let the tracking service generate its canonical ID. Timestamp collisions, human edits, and concurrent executions make self invented IDs fragile. 4. Use tags for filtering. Common tags might be team, stage, dataset version, model family, and evaluation protocol. Use a controlled vocabulary so one team doesn't log production while another logs prod. 5. Correlate across systems. Store the tracker run ID in pipeline logs and model registration metadata; store the pipeline job ID in the tracking run. A model candidate should be traceable back to the exact execution that created it. 6. Separate research from deployment status. An experiment run may produce a candidate without making that candidate production ready. Model approval and deployment are separate controlled processes. How should a team handle nested experiments? Suppose a hyperparameter search launches 10 trials . One parent run can represent the search request and 10 child runs can represent individual trials. The parent helps navigation, while the children own their metrics and parameters. If you log one run per fold, explain whether the reported summary metric is an aggregate across folds or a metric for one fold. Nested tracking structures are helpful but optional. Do not make a complicated hierarchy for a tiny study. Common mistakes Treating display names as guaranteed unique identifiers. Making one experiment per run or putting unrelated objectives in the same experiment. Reusing an old run ID for a distinct training attempt. Logging different evaluation protocols under one metric name without tags or an explanation. Losing the connection between an orchestration task and the resulting experiment run. Embedding credentials, private customer records, or sensitive identifiers in names or tags. Assuming a tracking run and a production model version are the same object. In short: Group comparable work into experiments , represent individual executions as runs , and use system generated immutable run IDs as canonical references. Consistent names, tags, and cross system correlation make experimentation searchable without compromising identity.
How do you log parameters, metrics, files, and model artifacts with MLflow?
MLflow logs the inputs, measurements, supporting files, and trained model produced by an ML run. Four common operations are: log params for configuration, log metric for measured values, log artifact for ordinary files, and a model flavor logger such as mlflow.sklearn.log model for a loadable model. Question 76 described what a tracking system should retain ; Question 77 organized experiments and runs. Here we focus on how to log each kind of output correctly with working Python code . Beginner friendly example A food delivery company wants to predict delivery duration from distance. It trains a simple regression model on historical orders, then records: Parameters: the training configuration, such as fit intercept=True. Metrics: validation mean absolute error (MAE) in minutes. Files: a plain text evaluation report. Model artifact: a saved scikit learn regression model that can later be loaded for prediction. Suppose the validation labels are 20, 30, and 40 minutes , while predictions are 22, 27, and 41 minutes . The absolute errors are 2, 3, and 1 minute , so: MAE = (2 + 3 + 1) / 3 = 2.0 minutes. That 2.0 is a metric, not a parameter. It is calculated from predictions and actual values, whereas fit intercept=True is a setting chosen before training. Complete minimal MLflow example Install the dependencies in an isolated environment: Then use this self contained demonstration: The example dataset follows an exact linear relationship, so the computed MAE may be extremely close to zero . The earlier 2.0 minute calculation is a separate hand worked metric example, not an expected result from this code. A production application would use real training data and a held out dataset. In recent MLflow versions, the name argument names the logged model . Older examples often use artifact path, but prefer the form supported by your installed MLflow version. What is the difference between the logging functions? Parameters: mlflow.log param("learning rate", 0.01) or mlflow.log params({...}). These describe the configuration of a run. Do not treat them as a time series or repeatedly overwrite the same parameter key with different values. Metrics: mlflow.log metric("val mae", 2.0). Metrics are numeric and can be logged repeatedly with steps for a learning curve: Log this loop inside an active run . Choose meaningful metric names and include units or record the definition and evaluation dataset. Files: mlflow.log artifact("evaluation.txt", artifact path="reports") uploads or copies one local file as a run artifact. mlflow.log artifacts("plots/") logs a whole directory. Files can include reports, charts, JSON summaries, and similar outputs. Models: mlflow.sklearn.log model(...) packages a trained model with MLflow model metadata and flavor specific loading information. A generic log artifact("model.pkl") saves a file but does not by itself create the same standardized MLflow Model format. For several frameworks, use the relevant flavor logger (for example mlflow.pytorch.log model) rather than assuming a scikit learn loader understands every model. How do you retrieve the logged model? A logged model's URI can be used to load the model: Run this in an environment that can access the MLflow tracking/artifact storage and has compatible dependencies. Saving a model does not automatically deploy it to a production endpoint. Operational checks 1. Start a run intentionally. A with mlflow.start run() context provides clear boundaries and records success or failure. 2. Log actual measured metrics. Do not log fabricated metrics as though a model computed them. 3. Check artifact availability. Confirm reports and models are retrievable, not merely present in the UI listing. 4. Avoid sensitive data. Do not put API keys, raw customer records, or private credentials in params, tags, or exported files. 5. Choose consistent metric names. validation mae minutes is more informative than an ambiguous score. 6. Use durable storage in teams. Local file:./mlruns is suitable for a demo; a shared tracking server and configured persistent artifact store are generally preferable for collaboration. 7. Record dataset/evaluation context. Reference input dataset identities and evaluation splits so the numeric metrics can be interpreted. Common mistakes Logging a hyperparameter as a metric and losing its fixed configuration meaning. Logging just the final score without the training setup. Saving a pickle as a generic file while assuming MLflow recorded model flavor metadata. Logging thousands of redundant metrics without a meaningful step or monitoring purpose. Comparing numerical metrics from different evaluation datasets without noting the difference (Question 79). Assuming tracking or model logging automatically guarantees reproducibility, quality, or production deployment. In short: Use log params for configuration , log metric for numeric results , log artifact for files , and a framework specific log model for reusable trained models . Verify the records and artifacts after the run ends.
How do you compare runs when datasets or evaluation protocols differ?
A lower error or higher accuracy does not automatically mean one ML run is better than another. When runs use different datasets, splits, labels, metrics, or preprocessing, first establish a common evaluation contract . Compare candidates on the same held out examples, with the same ground truth and scoring rules. Question 78 explained how to log experiment results. This lesson explains when logged results are actually comparable . Beginner friendly numerical example A food delivery company trains two delivery time predictors: Run A: MAE 4 minutes , measured on a small, easy urban delivery dataset. Run B: MAE 6 minutes , measured on a harder dataset covering traffic, bad weather, and long distance deliveries. It is tempting to conclude A is better because 4 < 6 . But that ranking is not justified : the runs were tested on different populations. The team now evaluates both models on the same four deliveries : Actual delivery times: 20, 30, 40, 50 minutes . Model A predictions: 24, 33, 46, 55 . Model B predictions: 22, 28, 43, 49 . Compute absolute errors: Model A: |24−20|, |33−30|, |46−40|, |55−50| = 4, 3, 6, 5 . MAE = 18/4 = 4.5 minutes . Model B: |22−20|, |28−30|, |43−40|, |49−50| = 2, 2, 3, 1 . MAE = 8/4 = 2.0 minutes . On this shared evaluation set, B has the lower MAE. Its improvement over A is (4.5−2.0)/4.5 × 100 ≈ 55.6% on these four cases. This is a toy example , not sufficient evidence of real world superiority: a production test needs a representative, sufficiently large evaluation set and uncertainty assessment. What must match for a fair comparison? 1. Dataset identity: Use the same immutable evaluation snapshot/version and confirm its contents, not just its display name. 2. Examples and labels: Evaluate against identical record IDs and ground truth, with compatible inclusion/exclusion rules and target definitions. 3. Split boundaries: Preserve train/validation/test membership; do not compare a model tested on training data against one tested on a truly unseen test set. 4. Metric definition: Compare MAE to MAE with the same units, weighting, aggregation, and missing value treatment. Accuracy and F1 are not interchangeable. 5. Preprocessing and postprocessing: Standardize how missing features, label decoding, thresholds, and output clipping are handled, unless the preprocessing is part of the model being fairly evaluated. 6. Evaluation protocol: Fix the benchmark version, scorer code version, business slices, time window, and any relevant subgroup/threshold policy. 7. Leakage controls: Ensure no test labels, future information, or duplicated entities leaked into model development. The training datasets may differ intentionally when studying whether extra training data helps; the evaluation dataset and scoring contract should still be held fixed. What if an identical evaluation set is unavailable? Label run results as not directly comparable ; present each with its own benchmark ID and context. Re evaluate both saved model artifacts on a newly agreed common holdout, if legally and technically possible. If one model only supports a subset of the population, report performance on the shared population and separate coverage and slice results. Do not silently exclude difficult cases. For time shifted datasets, use a deliberately designed temporal/out of time test, noting any changed business conditions. If evaluation conditions cannot be standardized, avoid claiming a single winner solely from the reported metric. Small Python comparison The code assumes both prediction arrays correspond to the same ordered records . In production, join predictions to ground truth by an immutable record key and check for missing/duplicated IDs before scoring. How should the tracking system help? Record, or attach to evaluation results: eval dataset id and an immutable eval dataset version or content identity; evaluation protocol version, scorer implementation commit, target units and metric definitions; split id, time horizon, data cutoff and subgroup/slice labels; model artifact ID and the run that generated the predictions. MLflow can associate tracked datasets with runs and supports searching by dataset names, digests, and contexts. A dataset digest's meaning depends on dataset type ; do not assume it is always a full cryptographic hash of every raw byte. See the official MLflow dataset tracking and run search documentation. A leaderboard or metric sorting screen is useful only after comparable runs have been filtered by their evaluation contract. Compare uncertainty, not only averages For a meaningful final decision, report performance on important slices (for example, short vs long distance deliveries), error distribution, and a confidence interval or a paired bootstrap if appropriate. A difference like 4.50 vs 4.49 minutes may be noise, whereas a stable paired improvement across representative deliveries is more convincing. Repeatedly selecting on the same held out test set can itself cause evaluation overfitting. Use validation data during development and reserve an appropriate untouched final test set for release decisions. Common mistakes Treating a lower MAE on an easier dataset as proof of a better model. Mixing minutes and seconds, macro and micro averaging, or differing class thresholds. Comparing different time windows without recognizing data drift. Reusing the same dataset name after its rows or labels change. Ignoring coverage differences and reporting only the easiest common subgroup. Using the final test set repeatedly for hyperparameter tuning. Comparing only point estimates without checking sample sizes and uncertainty. In short: First align the evaluation data, target, metric, split, and protocol . Then compare scores and uncertainty. If the conditions cannot be made comparable, say so rather than manufacturing a winner.
How do you capture the exact code commit and environment used for training?
To reproduce a training run, record the exact code revision and the environment in which that code executed. The strongest evidence is a full Git commit SHA, confirmation that the working tree was clean, a dependency lockfile identity, and an immutable container image digest. Attach these identifiers to the experiment run itself. Question 79 addressed fair evaluation across datasets and protocols. This lesson focuses on capturing the executable implementation and runtime , not comparing model scores. Beginner friendly practical example Two engineers train a delivery time model using exactly the same 100,000 order dataset and hyperparameters. Engineer A runs code from commit a81f... with scikit learn 1.x. Engineer B uses commit c92b... after a preprocessing bug fix and a changed dependency environment. Their model outputs differ. Without environment records, the team cannot tell whether the difference came from the code change, package versions, or other execution conditions. A clear run record would include: Git commit SHA: The exact committed training source, e.g. a full 40 character SHA 1 identifier in a repository using Git's SHA 1 object format (some repositories use SHA 256). Working tree state: clean, or explicit evidence of uncommitted changes. A Git SHA alone does not capture uncommitted edits. Dependency identity: A pinned lockfile plus its checksum or an archived copy. Container image digest: registry.example.com/train@sha256:..., rather than only a mutable tag such as train:latest. Runtime: Python, framework, CUDA/driver, OS/architecture, and training device details where relevant. Related run identifiers: The experiment run ID and any upstream pipeline or dataset reference. The content identifiers here are examples, not real repository or container IDs. Why Git commit alone is not enough git rev parse HEAD returns the commit currently checked out: If the second command returns lines, some tracked or untracked working tree changes may not be captured by the commit. Check which untracked files are actually used; Git ignore rules can hide generated source or configuration too. For reliable production training, a practical policy is fail a build when relevant source files have uncommitted changes , or capture a reviewed patch and other files as an explicit additional artifact. Merely setting a tag dirty=true signals uncertainty but cannot reproduce the changes by itself. Lockfiles and container digests A dependency list such as numpy =1.20 allows different versions to be installed on different days. Prefer resolved dependencies with exact versions and, where supported, package hashes, plus a checksum or archived copy of the actual lockfile. For example: The checksum identifies the lockfile's bytes. It does not prove the environment matches the file unless that lockfile was actually used and installation succeeded. A Docker tag like my training:latest may later point to a different image. A content addressed image reference such as: identifies a specific image or image index. For multi platform images, also capture the selected platform or platform specific manifest when exact image selection matters. A container digest improves repeatability but does not freeze host GPU drivers, attached devices, or nondeterministic computation. Step by step MLflow example This simplified snippet captures useful details in the active training environment: Important: This assumes Git, requirements.lock, and the TRAIN IMAGE DIGEST variable are actually available in the job. TRAIN IMAGE DIGEST must come from the deployment system that launched the container. This example demonstrates metadata recording; it does not install packages, build an image, or train a model. MLflow may automatically populate mlflow.source.git.commit when executed from a Git repository, but that automatic detection should be verified for your execution environment instead of relied on blindly. An explicit application tag can supplement it. A small numerical reason this matters Suppose Run A achieved MAE 4.2 minutes and Run B achieved 4.0 minutes , using identical evaluation data. The improvement is: (4.2 4.0) / 4.2 × 100 ≈ 4.76%. If the code commit, installed package versions, and runtime are unknown, the team cannot confidently explain or reproduce this difference. Capturing execution provenance makes a later investigation possible; it does not by itself prove which change caused the difference. What else affects exact replay? Dataset and feature identities: A code/environment record complements immutable input snapshots; it doesn't replace them. Randomness: Capture seeds and relevant deterministic settings. Hardware and libraries: GPU model, compute precision, CUDA/cuDNN, and driver versions may influence numerical results. Build provenance: Record the CI build identifier and image construction details when available. External dependencies: A model that fetches mutable web data or API results can change despite fixed code and container images. Secrets and credentials: Do not log their values. Record only non sensitive references to managed secret versions if appropriate. Even with identical code, data, and images, bit for bit identical training outputs may not be possible when algorithms or accelerators are nondeterministic. The purpose of provenance is to make conditions inspectable and replayable as closely as the system permits . Common mistakes Recording main or latest rather than a commit SHA or container digest. Logging the commit but forgetting local edits that affected training. Saving a requirements.txt with broad version ranges and assuming dependencies are frozen. Hashing a lockfile that was never used to build the actual runtime. Confusing an image digest with the full runtime—including host drivers and hardware. Saving code/environment metadata in a disconnected spreadsheet instead of associating it with the exact tracked run. In short: Capture Git SHA + source cleanliness + lockfile identity + immutable container digest + runtime details on each training run. This is the foundation for trustworthy execution provenance; the next lesson examines retaining those records during long running jobs.