MLOps

MLOps applies software engineering and operations discipline to machine learning. It covers reproducible training, model registries, deployment, monitoring, governance, and incident response.

What is configuration as code for an ML application?

Configuration as code means storing an ML application's adjustable settings in structured, reviewable configuration rather than scattering fixed values throughout Python scripts or notebooks. For example, the model's learning rate, input dataset reference, feature choices, evaluation threshold, and prediction timeout can be settings. The code explains how the application works; configuration chooses how it should run for a particular job or environment. A beginner friendly example Imagine a food delivery company training a model to predict delivery times. Initially, its training script contains fixed values such as learning rate = 0.01 and batch size = 64. To try different settings, an engineer must edit the script each time. That makes reviews noisy and complicates comparisons between experiments. Instead, the team creates a small, non secret YAML configuration file: The training code reads this file and uses the selected settings. To try a batch size of 128, the team changes the configuration or supplies a supported command line override— without rewriting the training algorithm . Here, MAE (mean absolute error) is the average absolute difference between predicted and actual delivery times. The max mae minutes setting is an example of a target the team checks before considering a model for release; it is not a universally correct threshold. How should a team manage configurations? 1. Separate code from choices. Keep reusable logic in code and adjustable values in clearly named configuration files, such as configs/train.yaml. 2. Version non secret configuration in Git. Review setting changes in pull requests alongside relevant code changes. 3. Validate values before running. Check required fields, expected data types, and sensible ranges. For example, a batch size should be a positive integer. 4. Track the actual values used. Save the resolved configuration with the training run record, along with its Git commit and dataset version. This matters if defaults or command line overrides changed the effective settings. 5. Allow safe overrides. Tools such as Hydra can compose configuration files and override parameters from the command line, which helps experiments stay organized. 6. Keep secrets separate. Store passwords, tokens, and service credentials in an appropriate secrets manager or securely injected environment variables—not in Git tracked YAML files or experiment logs. For instance, an experiment tracker such as MLflow can record the parameters used in each training run, allowing the team to compare results later. How is this different from Infrastructure as Code (IaC)? IaC defines resources the application runs on , such as cloud storage, network settings, and training compute. Configuration as code defines the application's behavior , such as which dataset version to read, which features to use, or what prediction timeout to enforce. They complement each other: IaC can provision a training service, while application configuration controls an individual training run. What about development, staging, and production? Some settings should differ between environments, such as endpoints and resource references. But avoid copying entire files unnecessarily; share common defaults and override only what truly differs. The next concept explores how to manage those differences safely. In short: Configuration as code makes ML settings visible, testable, versionable, and reproducible , while allowing one codebase to support many controlled experiments and deployments.

How should configuration differ across development, staging, and production?

An ML application often runs in three environments: development , staging , and production . The application code should remain as consistent as possible, but some settings must differ because each environment has a different job. Development (dev): A safe place to build features, explore ideas, and troubleshoot. Staging: A realistic rehearsal environment for checking a proposed release. Production (prod): The live system serving real users or business workloads. The goal is controlled differences , not three independently maintained applications. A practical example Imagine a food delivery service whose model predicts delivery times. The team needs to improve how the application handles restaurant preparation time data. In development , an engineer runs the prediction service against a small, synthetic dataset. Detailed debug logs are useful, and the service can use a low cost local or test endpoint. It must not accidentally modify live order data. In staging , the team deploys the same candidate application and model artifacts that it intends to release. The service uses an isolated test endpoint, representative appropriately sanitized or authorized data, and dependencies or capacity suitable for meaningful compatibility and performance tests. Staging should resemble production in the behavior being tested, even if its traffic volume or cost is lower. In production , the approved release uses live endpoints, stricter access controls, appropriate logging and alerts, and capacity sized for real demand. Changes go through the team's release and rollback process. What should differ between environments? 1. Data sources and destinations: Dev can use synthetic samples; staging uses isolated representative test data; production connects only to approved live data sources. Avoid letting tests write to production systems. 2. Service endpoints: Use separate URLs, databases, queues, storage locations, and service identities where isolation is needed. For example, staging must not send an experimental delivery estimate to a real customer. 3. Resource limits and cost: Dev may use smaller instances. Staging must have enough fidelity to test meaningful performance assumptions. Production capacity reflects availability and workload needs. 4. Logging and diagnostics: Dev may allow verbose debugging. Production needs structured monitoring and privacy safe logs; staging should check that those production monitoring paths work. Avoid logging personal data or secrets in any environment. 5. Access and change controls: Developers may have broad access to their own sandbox, but production permissions should be limited and audited. Production configuration changes usually need stricter approval. What should stay consistent? Keep these as consistent as practicable: Application code, dependency versions, and container image: Promote a tested immutable artifact rather than rebuilding different code for each environment. Feature definitions and model input/output schemas: Different feature calculations in staging and production can silently change predictions. Critical validation rules: Schema checks, input handling, and other correctness rules should not disappear in production. Configuration structure: Use the same names, expected types, and validation across environments, even when the values differ. There can be legitimate environment differences, but document them and test anything that might change model behavior. How can you implement this simply? A common approach is to keep shared, non secret defaults in version control and override only environment specific settings. For example: These files illustrate the idea; your application must explicitly load the base configuration and the selected environment file, and validate the result. The URLs are examples, not deployed services. Inject credentials using a secrets manager or another approved secure mechanism rather than committing them to YAML. The release pipeline should record the resolved configuration version alongside the application image, model version, and deployment environment. That makes it easier to answer, “What exactly was running when this prediction changed?” What is the most common mistake? Treating staging as an unrelated test system. If its software, feature definitions, or dependencies differ too much from production, passing staging tests may create false confidence. Aim for production like behavior with isolated resources , not necessarily an identical bill for compute. In short: Make dev easy to experiment in, staging realistic and isolated, and production controlled and observable —while promoting the same tested code and model artifacts through each environment.

How do you keep credentials out of source code and experiment notebooks?

A credential is something that proves an application or person is allowed to access a resource. In an ML project, that might be a database password, cloud access token, API key, or service account credential. Never type a real credential directly into a Python script, Git tracked configuration file, or notebook cell. Code is often copied, reviewed, shared, or published, and notebooks can save values in outputs as well as inputs. A practical example A food delivery company's ML team trains a delivery time prediction model using order history stored in a protected cloud bucket. A risky notebook might contain: If the notebook is committed to Git or shared with teammates, that token can leak. A safer design lets the notebook request access without containing the secret . How should credentials be handled? 1. Prefer managed identity or short lived credentials. On cloud platforms, give the training job or notebook environment an authorized workload identity or role. The cloud SDK can obtain temporary credentials without embedding a long lived key in code. Grant only the access the job needs. 2. Use a secrets manager when an actual secret is necessary. For example, a third party API may require an API key. Store it in an approved secrets service (such as AWS Secrets Manager), restrict who can retrieve it, and fetch it when the job runs. 3. Keep local developer secrets out of Git. A local .env file or environment variable can supply development credentials, but protect the file, add it to .gitignore, and commit only a placeholder .env.example. Environment variables reduce hard coding; they are not a substitute for access controls or secure secret storage. 4. Do not reveal secrets in notebooks or logs. Avoid printing tokens, dumping environment variables, including credentials in error messages, or saving output cells that expose sensitive values. Clear sensitive outputs and notebook checkpoints before sharing, and check exported HTML/PDF versions too. 5. Use secret aware CI/CD authentication. In automated jobs, prefer the CI platform's workload identity federation or another short lived identity mechanism rather than a permanent cloud key copied into repository settings. 6. Enable leak detection and rotation. Secret scanning and push protection can block some accidentally committed credentials. Restrict permissions and rotate secrets when required; scanners are additional defenses, not guarantees. A small Python example When an application genuinely needs a token supplied securely at runtime, it can read it from an environment variable: This code contains the variable name, not the secret value . The deployment system is responsible for securely supplying the value. For supported cloud services, managed identity is often better than handling a token at all. What if a secret is accidentally committed? Treat it as compromised, even if the repository is private: 1. Revoke or rotate it immediately with the credential provider; don't merely delete the line from the latest commit. 2. Check the provider's access logs for suspicious use and follow your incident response process. 3. Remove exposed values from current files, outputs, artifacts, and logs where possible; coordinate any Git history cleanup carefully, since old clones and forks may still contain the secret. 4. Add a preventive check so the same mistake is less likely to happen again. How is this different from configuration as code? A Git tracked training configuration can safely say dataset version: orders october or batch size: 64. It should not contain the password that grants access to the dataset. Configuration selects behavior; an identity or secret management system supplies protected access. In short: Use managed identities where possible, secret stores when needed, clean notebooks and logs, and rotate exposed credentials promptly. That lets ML teams collaborate without spreading access keys through their experiments.

What is environment parity and why does it matter for ML deployments?

Environment parity means keeping the important parts of development, staging, and production similar enough that an ML application behaves predictably as it moves between them. It does not mean copying all data, credentials, traffic, or machine sizes between environments. It means minimizing unintended technical differences that make a model work in testing but fail after deployment. A beginner friendly example Imagine a food delivery app that predicts delivery times. The model works in a developer's notebook, but after deployment the prediction service starts producing different estimates for the same example order. The team discovers that: The notebook and the service use different versions of a preprocessing library. The staging service receives a distance in kilometers , but the production integration sends meters . The developer's machine uses a different Python runtime from the deployed service. The model file may be unchanged, yet its inputs and results can differ. That is an environment parity problem . The distance mismatch is also an example of training serving skew : the features seen at prediction time do not match what the model expects. What should be kept consistent? Focus on components that affect correctness: 1. Code and model artifacts: Promote a tested, versioned application build and the intended model version. Avoid quietly rebuilding different code for each environment. 2. Runtime and dependencies: Pin and track relevant Python, package, system library, and container versions. A dependency upgrade should be deliberate and tested. 3. Feature processing: Use compatible feature definitions, units, schemas, and missing value handling at training and inference time. 4. Important services and APIs: Exercise compatible database, queue, object storage, or prediction service interfaces; replacing a production dependency with a very different local substitute can hide failures. 5. Hardware behavior when relevant: Check CPU/GPU availability, architecture, and accelerator/library compatibility. Hardware differences can affect speed and sometimes numerical results. These checks complement environment specific settings such as service URLs and access permissions; the earlier configuration topic explains why those settings legitimately differ. How can an ML team check parity? Use a short, repeatable release checklist: Build once and promote: Package code and dependencies in a versioned container image or similar reproducible artifact. Promote the same tested image by immutable digest through staging to production where possible. Test representative inputs: Send a small set of known examples through the candidate training/serving preprocessing paths and compare their feature values. Run integration tests in staging: Check actual model loading, input/output schemas, dependency interfaces, and prediction API behavior with isolated test resources. Verify deployment metadata: Record and compare the code commit, image digest, model artifact version, relevant dependency versions, and resolved non secret configuration. Run production smoke checks: After deployment, confirm the service starts, loads the approved model, and returns valid responses. Monitor subsequent errors and prediction quality. For floating point model outputs, allow an appropriate numerical tolerance when checking equivalent inputs. Different hardware or libraries can cause small numerical differences; the goal is to detect meaningful mismatches rather than promise byte for byte equality. Does staging need the same size as production? Not always. A small staging system may be sufficient for functional checks, but it cannot reliably prove production scale throughput or latency. Use appropriately sized performance tests when scalability matters. Keep real credentials and private production data isolated . In short: Environment parity makes ML releases more dependable by reducing hidden differences in software, model inputs, and execution. Build consistently, test equivalent behavior, and check for unexpected changes at deployment.

How can dependency versions be pinned in an ML project?

Dependency pinning means choosing and recording specific versions of the software packages an ML project uses, rather than allowing an installer to select a different version whenever someone sets up the project. This matters because a training script that runs today may behave differently—or fail—after a library upgrade. Pinning helps the team rebuild a known, tested software environment . A practical example Imagine a food delivery team training a model that predicts delivery times. The model's Python code imports numpy, pandas, and scikit learn. If the team only writes scikit learn in its dependencies, two engineers installing the project on different days might receive different versions. A newer version could change an API or the behavior of model loading. One straightforward approach is to record exact versions in requirements.txt: Then create a fresh environment and install them: The == operator asks pip for that exact package version. This is direct dependency pinning . How should a team pin dependencies? 1. List the packages the application needs. Maintain them in pyproject.toml or a requirements input file. For an installable Python library, compatible version ranges may be preferable; a deployable ML application often needs a separately resolved, exact environment. 2. Resolve and record the complete environment. Packages depend on other packages, called transitive dependencies . Pinning only three top level libraries does not guarantee that every dependency version stays the same. A tool such as pip compile can produce a fully pinned requirements.txt from requirements.in or pyproject.toml. 3. Recreate the environment from the recorded versions. Install the pinned requirements in a clean virtual environment or automated build, then run the project's tests. 4. Record the runtime constraints too. Python version, operating system, CPU/GPU architecture, and accelerator libraries such as CUDA can affect which packages install and how they run. Python package pins alone do not freeze these components. 5. Update deliberately. Check for security fixes and compatibility changes, upgrade selected packages, rerun training or inference checks when needed, and commit the updated dependency record after review. For a small experiment, python m pip freeze requirements.txt captures package versions already installed in the active environment. However, if that environment contains unrelated packages, the output captures those too. A clean environment and a dependency resolver are usually better for a maintained project. What does pinning actually guarantee? A line such as scikit learn==1.6.1 constrains which version of that named package pip installs. It does not , on its own, guarantee: The same versions of the package's dependencies. Identical Python, operating system, or GPU environments. The same training data, random seeds, or resulting model. That an old dependency is secure forever. For stronger repeatability, teams also record the resolved dependency set, runtime details, input data and training settings. Build hashes and verified package sources can further reduce supply chain risks. How is this different from environment parity? Pinning records specific software versions. Environment parity is the broader goal of making development, staging, and production behave consistently. Pinning is one tool for achieving that goal, not the whole solution. The next concept explains lockfiles , which record a resolved set of dependencies more systematically. In short: Pin and test the package versions your ML application relies on, capture indirect dependencies, and update them intentionally. This makes failures easier to diagnose and model releases easier to reproduce.

What are lockfiles and why are they useful in reproducible ML environments?

A lockfile is a machine generated record of the exact software dependencies chosen for a project. It helps different developers, automated tests, and deployment systems install the same resolved package versions instead of selecting new versions independently. In the previous concept, we learned how to pin dependencies. A lockfile builds on that idea: it captures not only the packages you requested, but also the indirect packages those packages need. A practical example Imagine a food delivery team that trains a model to predict delivery times. Its project declares that it needs pandas and scikit learn. These are direct dependencies . Each may need additional libraries—for example, scikit learn depends on packages such as NumPy and SciPy. These additional packages are transitive dependencies . If developers install from broad version requirements on different dates, the installer may resolve different compatible versions of those indirect libraries. The training script could then behave differently or fail. A lockfile records the resolved dependency set so the team can recreate the tested installation more reliably. What is the difference between a dependency declaration and a lockfile? Dependency declaration (such as pyproject.toml): States what the project needs , often using allowed version ranges. Lockfile (such as uv.lock or poetry.lock): Records which versions were selected for the dependency graph, with additional metadata that depends on the format. Think of a dependency declaration as a shopping list and a lockfile as the exact order that was placed—including all the supporting ingredients. A small Python example using uv In an existing uv managed project, the team might run: Here: 1. uv add records the declared dependencies in pyproject.toml and updates the project's lockfile. 2. uv lock ensures the project's uv.lock is generated or updated. 3. uv sync locked installs the locked environment only if the lockfile is already consistent with the project requirements . Otherwise it fails rather than silently changing the lockfile. The team commits both pyproject.toml and uv.lock to Git. The same command can be used in CI/CD so a proposed change is tested against the recorded dependency resolution. Other tools use similar concepts: Poetry has poetry.lock; the Python packaging ecosystem also specifies a tool neutral pylock.toml format. These are different formats, so use an installer that supports the one your project produces. Why are lockfiles valuable in MLOps? 1. Consistent environments: Teammates and CI jobs use the versions selected when the application was tested. 2. Easier debugging: If a new dependency resolution breaks model loading, the team can compare the lockfile changes to see what changed. 3. Controlled upgrades: Libraries change when the team intentionally updates the lockfile, reviews the difference, and reruns tests. 4. Release traceability: A training run can record its Git commit and lockfile revision alongside the dataset, configuration, and model version. 5. Less guesswork during deployment: The build process can install a recorded dependency set instead of solving version choices from scratch. What does a lockfile not guarantee? A lockfile is a big step toward reproducibility, but it does not guarantee identical ML predictions on its own. Runtime and platform still matter: Python version, operating system, CPU/GPU architecture, CUDA drivers, and availability of compatible package builds can affect installation and behavior. Some lockfiles represent multiple supported environments. Model inputs still matter: The same packages with different training data, random seeds, or training settings may produce a different model. Security still matters: A locked version can develop a known vulnerability. Scan dependencies and refresh them through a tested update process. Integrity depends on the format and installer: Some lockfiles record package hashes and other artifact details, but don't assume every lockfile or install workflow offers the same guarantees. In short: Pinning a few packages restricts version choices; a lockfile records a resolved dependency set. Commit it, install from it in CI and deployments, and update it deliberately to make ML environments more consistent and easier to diagnose.

How do virtual environments and containers solve different problems?

A Python virtual environment and a container both help an ML application run with fewer surprises, but they isolate different things . Virtual environment (venv): Gives a Python project its own installed Python packages, separate from other projects on the same machine. Container: Packages an application with its runtime and supporting user space system libraries and runs it as an isolated process. A container generally shares the host's operating system kernel; it is not a complete virtual machine. The important distinction is: a venv isolates Python dependencies; a container provides a more complete, portable application runtime. A practical example Imagine a food delivery company training a model to predict delivery times. One developer works on the forecasting model with one version of scikit learn. Another project on the same laptop requires a different version. A venv lets each project have its own Python packages so an upgrade for one project does not unexpectedly affect the other. Later, the delivery time model works on that laptop but fails on the deployment server because the server has different operating system libraries and startup settings. A container image can package the application code, chosen Python runtime, required Python packages, and supporting system libraries together. The team can build and test that image, then deploy the same image to a compatible environment. 1. When is a virtual environment enough? For local development or a small Python only training script, a venv is often the simplest choice: This creates an isolated place for project packages such as numpy or scikit learn. The packages installed in it normally do not change those of other virtual environments. However, a venv uses a base Python installation and depends on the surrounding machine for operating system facilities. It does not package a different OS, isolate system processes, or ensure that CUDA drivers and other external dependencies match. Recreate the venv on another machine rather than copying its directory. 2. When is a container useful? For a deployed prediction API, scheduled training job, or automated CI/CD workflow, a container often makes the runtime easier to distribute. A simplified Dockerfile might be: The image contains the selected Python runtime and installed packages, plus application files. The example assumes predict.py and requirements.txt exist. In production, the team should also select a reviewed, preferably digest pinned base image, handle image security, run with least privilege, and avoid copying local secrets or virtual environments into the image. A container is not fully independent of its host . CPU architecture, kernel compatibility, and—when using GPUs—host driver and accelerator support still matter. 3. Can you use both? Yes. Many teams use a venv while developing locally and a container for CI or deployment . Inside a container, a separate venv is often optional for a simple single application image, although some tools and workflows choose to use one. Neither tool decides which package versions are installed: you still need reviewed dependency specifications and, where appropriate, a lockfile. 4. Which should you choose? Exploring a model or running notebooks locally? Start with a virtual environment. Running a consistent prediction API across machines? Prefer a tested container image. Training on GPU infrastructure? Consider the container's libraries and its compatibility with host GPU drivers. Need an identical dataset or training result? Neither alone is sufficient; track input data, model artifacts, training settings, and relevant randomness as well. In short: A virtual environment prevents Python package conflicts; a container helps deliver a complete, repeatable application runtime. They solve related problems at different layers, and ML teams often use them together.

What is an immutable artifact in an MLOps workflow?

An immutable artifact is a saved output that is treated as unchangeable after it is created . If its contents need to change, the team creates a new artifact or version rather than silently replacing the old one. In MLOps, common artifacts include a trained model file, a container image used to serve predictions, an exported preprocessing component, or a versioned dataset snapshot. The purpose is simple: a version that worked yesterday should still refer to the same content tomorrow. A practical example Imagine a food delivery company that predicts delivery times. The team trains a model, evaluates it, and approves delivery time model version 12 . Its serving container is also identified by a specific image digest. A week later, new order data arrives. An engineer trains a better model. Instead of replacing the file behind version 12, the team registers version 13 . Now they can: Compare versions 12 and 13 fairly. Know which exact model produced a historical prediction. Deploy version 13 after approval without losing version 12. Restore version 12 if the new version performs poorly. If the contents of version 12 were quietly overwritten, its evaluation results and rollback record would no longer describe the same model. How do you recognize an immutable artifact? An artifact needs a stable identity tied to fixed content . Examples include: 1. A container image digest: An image reference such as registry.example/app@sha256:<digest identifies image content. Unlike ordinary tags such as latest, the digest is content addressed. (The placeholder is illustrative, not a real digest.) 2. A registered model version: A model registry such as MLflow assigns versions, for example models:/delivery time/12. The registry points to the model's recorded artifact. Reliable immutability also requires protecting the backing artifact store from replacement or deletion. 3. A versioned dataset snapshot: A specific dataset version or content hash lets a training job read the same input snapshot later, provided that version is retained. These are related but different: a version number names a record; a cryptographic hash identifies content . A trustworthy system should make sure versioned records continue to reference the intended files. What is the difference between a version and an alias? An alias such as champion or a container tag such as latest is usually a movable pointer . For example: Today, champion points to model version 12. After review, the team changes champion to version 13. Versions 12 and 13 remain separately identifiable. Aliases are useful for managing deployments, but record the exact resolved model version when logging or approving a release. Do not assume a mutable alias will always identify the same content. How should an MLOps team use immutable artifacts? A simple workflow is: Build → Test → Store as a new version → Record the exact identity → Approve → Deploy In practice: Record the Git commit, dataset version, training configuration, model version, and container image digest in the release record. Test the actual artifact intended for production rather than rebuilding it differently for each environment. Restrict permissions so approved artifacts cannot be casually overwritten; use storage versioning or immutability controls where needed. Retain older approved artifacts for the agreed rollback period and apply deliberate cleanup policies afterward. Important: Calling something “version 12” does not automatically make its storage immutable. Access controls and artifact retention rules must enforce the desired guarantee. Also, immutable artifacts do not guarantee identical predictions across incompatible runtimes or changing input data. In short: An immutable artifact is a fixed, identifiable output that can be tested, traced, reused, and rolled back to. A new model or image should get a new identity instead of changing an already approved artifact.

Why should production ML jobs be idempotent?

Idempotent means that running the same operation again with the same logical request does not create additional unwanted changes . This matters in production MLOps because scheduled pipelines and distributed workers can fail, time out, or receive duplicate requests. A task may run once, partly finish, and then run again. The second attempt should not duplicate data, publish extra model versions, or send the same notification twice. A beginner friendly example Imagine a food delivery company that predicts how many orders each neighborhood will receive tomorrow. A daily job: 1. Reads yesterday's completed orders. 2. Calculates features for each neighborhood. 3. Writes a demand forecast into a database. Suppose the job writes forecasts for half the neighborhoods, then loses its database connection. The scheduler retries it. Without idempotency: The retry simply appends new rows. Neighborhoods processed before the failure now have duplicate forecasts. A dashboard might double count their expected orders. With idempotency: Each forecast has a stable logical identity, such as forecast date + neighborhood + model version . Repeating the write safely replaces or recognizes the existing result instead of adding an unintended duplicate. For example, if the database has a unique constraint for those fields, an upsert (insert or update) can update the same logical row on a retry. The write should also be part of a suitable transaction or staging and publish process so users do not see a half completed daily result. How can an ML team make a job idempotent? Identify the work consistently. Use a stable key representing the intended logical run, such as pipeline name + input dataset snapshot + processing date. Retried attempts reuse that key; an intentionally new experiment gets a new identity. Read a fixed input snapshot. Do not silently switch from yesterday's data to the newest data when a job retries. Use a specified data partition or dataset version. Make output writes retry safe. Prefer upserts, unique constraints, or writes to a well defined output partition over blindly appending duplicate rows. Publish complete results. Write intermediate output to a temporary or staging location, validate it, and only then mark it as ready for downstream consumers. Protect external side effects. Register a model, update a deployment pointer, or send a notification only after checking whether that logical action already succeeded. Where possible, use atomic idempotency records and appropriate concurrency controls, not just an unsafe check then insert. Test retries deliberately. Interrupt a job after a partial write, restart it, and confirm that the final visible result is correct and contains no duplicate records. A simple mental model is: Same logical request → One correct final result, even after retries. Does idempotency mean model training produces identical weights? No. Some training algorithms use randomness or hardware operations that can cause small or meaningful differences between separate training runs. Idempotency focuses on safe, repeated execution and its visible side effects , not on promising that two independent training runs produce byte identical model files. For an expensive training task, the team may record a unique logical training run ID and reuse a successfully completed artifact after a retry. If it must rerun failed training, it should ensure a new artifact is not mistakenly published as a duplicate release. Idempotency versus immutability The previous concept explained immutable artifacts : once an approved artifact is created, its contents should not silently change. Idempotency answers a different question: What happens if the operation that creates or publishes an artifact is attempted more than once? They work together. An idempotent publication operation can return the same existing immutable artifact on a retry rather than creating an unintended second release. In short: Production ML jobs should be idempotent so that failures and automatic retries do not corrupt datasets, duplicate forecasts, or create inconsistent model releases.

How do you make an ML application twelve-factor friendly?

A twelve factor application follows a set of practices for making software easier to deploy, operate, and scale. The original Twelve Factor App methodology was written for services, but many of its principles also help production ML systems. In MLOps, the aim is not to force all model training into a web service design . Instead, apply these principles especially to model serving APIs, background workers, and the processes that deploy them. A practical example A food delivery company has trained a model that predicts delivery time. It wants to run a reliable prediction API. The first version works only on an engineer's laptop: it loads a model from a personal folder, writes predictions to a local CSV file, and needs someone to restart it manually. If a second server starts, it cannot find the same model or prediction history. A twelve factor friendly design makes each API instance replaceable: the model is fetched by a specific approved version , durable results go to a database, runtime settings come from the deployment environment, and the platform handles logs and restarts. Applying the twelve factors to the ML service 1. Codebase — One version controlled codebase, many deployments. Keep the prediction service in Git. Development, staging, and production deploy reviewed revisions of that codebase. Related training services may have their own codebases. 2. Dependencies — Declare and isolate what the service needs. Record Python libraries and required runtimes. Build the service with its tested dependencies rather than relying on whatever happens to be installed on a particular server. 3. Config — Keep deployment settings outside the code. Set values such as the approved model version, artifact location, and request timeout through the deployment's configuration mechanism. Supply sensitive credentials using an approved secret or workload identity system. For instance, the application can read non secret settings at startup: The serving code can then use these values to retrieve the approved, fixed model artifact . Neither the model file nor credentials need to be hard coded into the source. 4. Backing services — Treat storage and databases as attached resources. A model registry, object store, database, and message queue are external services reached through configured endpoints. Replacing a test store with an approved production store should not require editing the prediction algorithm. 5. Build, release, run — Keep these stages separate. Build a tested container or application artifact, combine it with environment specific configuration to form a release, and then run it. Do not silently install new packages or train a new model every time the prediction server starts. 6. Processes — Keep the serving process stateless. An API instance may cache the loaded model in memory for speed, but it must not depend on local disk or process memory for durable predictions, user sessions, or other shared business state. Store those externally. 7. Port binding — Expose the prediction API through a service port. A Python API could listen on a configured port while the deployment platform routes requests to it. 8. Concurrency — Scale using replaceable processes. If traffic grows, run additional prediction service replicas behind a load balancer. For expensive GPU models, measure memory use, batching, and throughput: blindly adding processes may exhaust accelerator memory. 9. Disposability — Start and stop safely. Make startup and health checks explicit. On shutdown, stop accepting new work, allow in flight requests to finish within a deadline, and let queue workers safely requeue or checkpoint unfinished work. Very large models can take time to load, so realistic readiness checks matter more than promising instantaneous startup. 10. Dev/prod parity — Test production like behavior. Verify the candidate image, model version, input schema, and critical service integrations before release; use isolated non production data and credentials. 11. Logs — Emit event streams. Write structured, privacy safe request and error logs to standard output/error and let the runtime platform collect them. Avoid storing the only copy of logs on a container's local filesystem. 12. Admin processes — Run maintenance as separate tasks. Database migrations, backfills, and one time model index updates should be controlled jobs using the appropriate application release, not ad hoc changes inside a running prediction API. What about long running ML training? Training is often intentionally stateful while it runs . It holds model weights and optimizer state in memory, perhaps for hours. Twelve factor thinking is still useful: place durable checkpoints, datasets, and final models in external versioned storage so a worker can be restarted or replaced safely. The earlier lessons on configuration, environment parity, immutable artifacts, and idempotent retries explain those individual controls. Twelve factor design brings them together into an operating model for a deployable ML service . In short: Make your serving code versioned, its dependencies explicit, its runtime configurable, its durable state external, and its processes replaceable. That turns a laptop only prediction script into a service that is easier to test, scale, and recover.