MLOps

MLOps applies software engineering and operations discipline to machine learning. It covers reproducible training, model registries, deployment, monitoring, governance, and incident response.

What is an MLOps reference architecture?

An MLOps reference architecture is a reusable blueprint showing the main parts needed to build, release, and maintain machine learning models in production. It is not one fixed set of tools. A small company and a large bank may choose different technologies, but both need to answer similar questions: Where does data come from? How is a model trained and approved? How do applications use it? And how will the team notice problems? A practical example Imagine a food delivery app that predicts how many orders may arrive in each neighborhood during the next hour. The prediction helps the company decide where to position delivery partners. A reference architecture helps the team connect the work into one reliable system instead of passing files around manually. Main building blocks 1. Data sources and storage: Collect order history, restaurant activity, weather, and other relevant data. Keep the raw or prepared data in a managed location. 2. Data validation and preparation: Check for missing records or unexpected values, and prepare the information needed for training. For example, an hourly order count should not accidentally include duplicate orders. 3. Training pipeline and experiment tracking: Run repeatable training and evaluation steps, while recording which code, data, settings, and results belong to each experiment. 4. Model registry: Store approved candidate model versions with their metadata so the team can identify exactly what is being considered for release. The registry is distinct from the place where datasets are stored. 5. Release automation and model serving: Use tests and an approval step before deploying a version. The model might serve predictions through an API, or a scheduled job might write forecasts to a database for the app. 6. Production monitoring and feedback: Watch prediction quality when actual order counts become available, along with data quality, errors, latency, and resource usage. Investigate problems and retrain when there is a good reason. How do the components connect? Data sources → Data checks and preparation → Training and evaluation → Model registry → Tested deployment → Predictions → Monitoring Monitoring findings feed back into the next data or model improvement cycle. The training pipeline and the prediction service are related, but they are usually separate running processes : the app should not have to retrain a model every time it needs a prediction. Two supporting systems operate across these stages: Source control and CI/CD: Track changes to code and pipeline definitions, run automated checks, and help deliver approved changes safely. Security, access, and traceability: Restrict who can read sensitive data or release models, and retain enough records to investigate how a production prediction system was built. Does every MLOps system need all these tools? No. This is a reference architecture , not a shopping list. For a small forecasting project, a scheduled training job, versioned model files, basic tests, and simple monitoring may be enough. Larger systems may add pipeline orchestration, feature stores, separate staging environments, automated rollbacks, and dedicated governance services. The important thing is to define each responsibility and make the connections dependable before adding complexity. In short: An MLOps reference architecture shows how data, training, model versions, releases, predictions, and monitoring fit together so an ML system can be operated and improved safely.

How do you choose between managed MLOps services and a self-hosted stack?

When building an MLOps platform, a team has an important choice: use a managed service or operate the MLOps tools themselves . Managed MLOps service: A provider operates much of the platform infrastructure. Examples include Amazon SageMaker AI and managed ML platforms from cloud providers. Your team still owns its data, model quality, application logic, access decisions, and suitable security settings. Self hosted MLOps stack: Your team installs and operates tools such as MLflow or Kubeflow, either on its own servers or on rented cloud infrastructure . You have more control, but you must handle more setup, upgrades, reliability, and support. Neither option automatically makes your models more accurate. A practical example Imagine a small online store building a model to forecast next week's product demand. The team has two ML developers and needs to launch a reliable forecast service soon. Option A — Managed: They use a cloud managed training and deployment service. The provider handles much of the underlying training or serving infrastructure, so the team can spend more time preparing data, testing forecasts, and connecting predictions to the store. Option B — Self hosted: They operate their own MLflow tracking server and training jobs, potentially on Kubernetes. They can customize how everything connects, but somebody must also maintain the services, credentials, storage, backups, and upgrades. For this small team, a managed platform may be the simpler starting point. Now imagine the company grows, already has a platform engineering team, and needs custom scheduling across several environments. Self hosting some components might become worthwhile. Six things to consider before choosing 1. Team skills: Can engineers reliably operate servers, containers, databases, and monitoring? If not, a managed platform usually reduces the operational burden. 2. Time to launch: Managed services generally help teams get started faster. Self hosting gives more choices but requires integration and maintenance. 3. Control and customization: Self hosting can be useful when the team needs unusual infrastructure, integrations, or deployment behavior. Check whether managed services already support those needs. 4. Security and compliance: Both approaches can protect sensitive data, but responsibilities differ. Confirm data location, network access, audit requirements, and who handles each security control. 5. Total cost: Compare more than service prices. Include compute, storage, idle resources, engineering time, upgrades, incident response, and support. Self hosted software may be free to download without being free to operate. 6. Portability and vendor dependence: Consider how easily you can move training code, model artifacts, and pipelines later. Open tools can help, but self hosting does not automatically guarantee portability. Can you combine them? Yes. Many teams choose a hybrid approach . For example, they might use open source MLflow for experiment tracking while using managed cloud jobs for model training and serving. The best design is the one your team can operate safely and sustainably , not necessarily the one with the most tools. A simple decision rule Lean team, fast delivery, limited platform experience: Start by evaluating managed services. Strong infrastructure team, specialized requirements, need for deeper control: Evaluate self hosting. Different needs across the lifecycle: Consider mixing managed and self hosted components. In short: Managed MLOps usually trades some infrastructure control for less operational work. Self hosting offers more flexibility but makes your team responsible for more of the platform. Choose by comparing your requirements, skills, and total operating cost.

How should an ML project be organized into repositories and packages?

A good ML project structure makes it easy for someone else to understand, test, and run your work. Instead of leaving everything inside one notebook, organize the project so experiments, reusable code, configuration, and tests have clear places. Two words are worth distinguishing: Repository: The complete project tracked in a version control system such as Git. It can contain source code, documentation, tests, and configuration. Package: Reusable Python code that other parts of the project can import. A package is usually just one part of the repository. A practical example Imagine a food delivery app that predicts how many minutes an order will take. You start by exploring order data in a notebook. Later, an automated training job and a prediction service both need the same feature processing code. Copying that code into three separate files would make updates risky. It is better to put the reusable logic in a Python package and let the notebook, training job, and service import it. A small project might look like this: Here's what belongs where: README.md: Explain what the model does, how to set up the environment, and how to run training and tests. pyproject.toml: Describe how the Python project is built and its dependencies. A lockfile can record exact resolved dependency versions when your environment tool supports one. configs/: Keep changeable settings—such as dataset paths, training options, or evaluation thresholds—separate from application code. Do not store passwords here. notebooks/: Explore data and try ideas. Once a step becomes important for repeatable training, move that logic into reusable Python code. src/delivery time/: Place the project's importable code here. For example, features.py computes distance or time of day features that training and prediction both need. tests/: Check important behavior automatically. For example, test that a missing restaurant preparation time value is handled as expected. pipelines/: Define the order of automated steps, such as validate data → prepare features → train → evaluate. The src/ layout helps distinguish installed application code from other repository files. For Python development, you can typically install a correctly configured project in editable mode with python m pip install e ., then import its package without copying files around. Where should data and trained models go? Small, non sensitive sample files may belong in the repository. Large or private datasets and trained model artifacts usually should not. Instead, store them in appropriate data storage, an artifact store, or a model registry. Use versioned references and metadata so the team can identify which data and model belong to a particular experiment or release. A .gitignore file can help prevent accidental commits of generated files, but it does not replace access controls or secret management. One repository or several? Start with one repository when one small team owns the training pipeline and prediction code. This keeps changes easy to coordinate. Separate repositories may make sense later if, for example, several teams share a stable feature processing library, or the serving application has its own release cycle and ownership. In that case, publish shared code as a versioned package rather than manually copying it between projects. Do not split the project just to look more sophisticated. Choose boundaries based on who owns the code, how often it changes, and how it is released. In short: Keep the repository easy to navigate, move reusable logic out of notebooks into a Python package, separate configuration and tests, and store large data and model artifacts in systems designed for them.

What is the role of Git in collaborative ML engineering?

Git is a version control system that helps a team record changes to project files and work on the same code without accidentally overwriting each other's work. In machine learning, this matters because a small change to feature processing code, training settings, or a prediction API can change how a model behaves. A practical example Imagine three engineers building a delivery time prediction model for a food delivery app: Priya improves the code that calculates delivery distance. Arun fixes a problem in the model training script. Meera adds tests to check that predictions never return a negative number of minutes. If everyone edits the same shared files directly, one change could accidentally replace another. Git lets each person develop a change separately, review it, and then combine the approved work. How a team uses Git 1. Start with a repository. The team keeps training scripts, reusable Python modules, tests, and configuration files in a shared Git repository. 2. Create a branch. Priya creates a branch called feature/delivery distance. A branch is a separate line of work, so she can make changes without immediately changing the team's main branch. 3. Commit the changes. Once her distance calculation and tests are ready, she records them in a commit —a named point in the project's history with a message describing the change. 4. Push and open a pull request. She shares the branch with the team through a hosting platform such as GitHub. A pull request asks teammates to review the proposed changes before merging them. 5. Review and test. Another engineer checks the logic, automated tests run, and the team discusses any issues. 6. Merge into the main branch. After approval, the change becomes part of the shared code. The team can see which commit introduced it. A simplified collaboration flow is: Create branch → Change code → Test → Commit → Push → Review → Merge Git records what changed and when . A hosting service like GitHub provides collaboration features such as pull requests and review discussions. Why is Git especially useful for ML projects? Traceability: When a model behaves differently, the team can inspect changes to its training or feature processing code. Safer collaboration: Separate branches allow engineers to test ideas without immediately affecting shared work. Reproducible code: A training run can record its Git commit identifier so others know which code version was used. Automated quality checks: Teams can run tests when changes are proposed, catching errors before they reach the main branch. Controlled releases: A team can identify which version of the code was used for a deployment and investigate or revert problematic changes. What should not go into Git? Git is very useful for code and small text based configuration files , but normal Git repositories are not a good place for large training datasets, huge model files, passwords, or API keys. For example, if the team trains a delivery model on millions of orders, it should store the dataset in appropriate protected storage. A tool such as DVC (Data Version Control) can track dataset versions through small metadata files committed to Git, while the actual data remains outside the repository. Model registries or artifact stores can serve a similar purpose for trained models. Also remember: Git alone does not guarantee reproducible ML results. Teams must record the relevant dataset version, environment, configuration, and other training details as well. In short: Git helps ML engineers collaborate safely, review code, and understand how a project changed. Combined with data and model versioning, it becomes an important foundation for reliable MLOps.

How should Git branches and pull requests be managed for ML projects?

A good branching and pull request policy lets an ML team make changes without accidentally breaking training pipelines or production predictions. The goal is not to create complicated Git rules; it is to make important changes easy to review, test, and release safely . If you are new to Git, first learn what branches, commits, and pull requests do. Here, we'll focus on the rules a team should follow . A practical example Imagine a food delivery app that predicts delivery times. An engineer wants to improve how the model calculates travel distance. The updated code passes basic Python tests, but it could still change the model's inputs or make delivery estimates worse. The team therefore needs checks that go beyond asking, "Does the code run?" 1. Keep the main branch protected Use main for reviewed, tested code. A practical starting policy is: Make changes in short lived branches, such as feature/better distance or fix/missing distance. Require a pull request (PR) before merging into main. Require at least one relevant reviewer and passing automated checks. Block direct pushes, force pushes, and accidental deletion of the protected branch unless a carefully controlled exception is necessary. Require code owner reviews for sensitive files if different teams own training, serving, or infrastructure code. On GitHub, branch protection rules or repository rulesets can enforce these safeguards. Your team can adjust the strictness as it grows. 2. Make each pull request easy to understand Prefer a small PR with one clear purpose rather than mixing a new feature, unrelated refactoring, and deployment changes. For the distance calculation update, a useful PR description answers: Why? Current distance estimates ignore a route condition and may be inaccurate. What changed? The distance feature calculation and its unit tests. What could be affected? Model training inputs and the prediction service. How was it tested? Unit tests passed; a representative evaluation dataset was checked. What happens if it fails? The previous code and approved model can remain in service or be restored. A PR template makes it easier for reviewers to find these details. 3. Check ML specific risks before merging Besides normal code quality and security checks, review changes that might affect model behavior: Data and feature contracts: Are column names, types, missing value rules, and feature calculations compatible with training and serving? Reproducibility: Can the run be traced to a code commit, configuration, and dataset or artifact version? Evaluation: If the model or its features change, was a relevant model quality comparison performed? A passing unit test does not prove prediction quality. Data safety: Does the PR avoid committing private records, secrets, and large model binaries? Production impact: Could the change increase prediction latency, resource usage, or deployment risk? For example, a PR that renames distance km to travel distance km might pass code style checks but break a prediction service still expecting the old field. A feature schema compatibility test can catch the problem. 4. Automate the right checks Run quick checks on ordinary PRs: formatting, linting, unit tests, dependency or secret checks, and simple pipeline validation. When a PR changes training logic or model features, also require appropriate integration tests and evaluation results. Full training can be expensive, so teams may run lightweight checks for every PR and reserve costly evaluations for high impact changes or controlled workflows. If a required check fails, fix it before merging. Approval should not be treated as a replacement for the required tests. 5. Separate merging code from releasing a model Merging a PR does not automatically mean a newly trained model is approved for production. After the code is merged, a training pipeline may produce a candidate model. That model still needs to meet the project's evaluation, approval, and rollout requirements. Record which Git commit and training artifacts produced the released version so the team can investigate problems later. In short: Protect the main branch, keep PRs focused, review both code and ML behavior, require meaningful automated checks, and give model releases their own approval process.

What should be committed to Git and what should be stored externally in an ML project?

Git is best for tracking the files a team needs to review, change, and reproduce its work . It is usually not the right place to store large datasets, trained model files, private customer information, or passwords. The useful rule is simple: keep the instructions for building the model in Git; keep heavy or sensitive assets in storage designed for them. Save references to those assets so the team can find the exact versions later. A practical example Imagine a food delivery company training a model to predict delivery times. Its project contains Python code, training settings, millions of order records, and a trained model file. Here's how to organize those files. 1. Commit these files to Git Source code: src/features.py, src/train.py, and src/predict.py. These explain how data is prepared, a model is trained, and predictions are calculated. Tests: Files in tests/ that verify important behavior, such as how missing delivery distances are handled. Non secret configuration: A configs/train.yaml file containing settings such as the selected features or training parameters. Keep passwords out of it. Environment definitions: pyproject.toml, dependency lockfiles, and other reproducible setup instructions. Documentation and automation: README.md, pipeline definitions, and CI workflow files. Small, safe examples: A tiny synthetic order dataset that lets new developers run a test without accessing real customer records. Version references: Small metadata files, such as DVC's .dvc files or dvc.lock, that identify data versions stored elsewhere. These files are usually small enough to review in a pull request. 2. Store these outside ordinary Git Real training data: Millions of customer orders belong in appropriately secured object storage, a data platform, or a managed dataset store—not in the code repository. Large trained models: Put model weights and other substantial artifacts in an artifact store or model registry. Record which version was used for a release. Secrets and personal information: API keys, database passwords, private customer records, and service credentials require approved secret management or access controlled storage. Never commit them. Generated outputs: Temporary logs, caches, experiment outputs, and intermediate datasets should go to suitable storage or observability systems when they need to be retained. Local environments: Folders such as .venv/ and pycache / can be recreated and do not belong in the repository. For large non sensitive files that really need Git linked versioning, Git LFS can be an option. For datasets that change frequently, tools such as DVC can track version metadata in Git while keeping the actual data in external storage. What does this look like in practice? The developer commits the training code and its configuration, but uploads the full order dataset to protected storage. A DVC metadata file, or another versioned dataset reference, identifies which data snapshot the training job should use. Later, if a prediction changes, the team can connect the Git commit , dataset version , training configuration , and model artifact version instead of guessing which files produced the model. How do you stop files being committed accidentally? A .gitignore file can exclude local files, for example: Commit the .gitignore itself. You can also commit .env.example containing placeholder variable names, not real credentials . Before committing, check the staged files and use secret scanning where available. Important: Adding a file to .gitignore does not remove it from Git if it was already tracked. If a secret was committed, revoke or rotate it immediately and follow your organization's incident and repository cleanup process. In short: Commit small, reviewable code, settings, tests, and version references. Store large, generated, and sensitive assets separately, then connect everything with traceable version identifiers.

Why are large training datasets unsuitable for ordinary Git storage?

Git is designed to track changes to project files, especially source code and other text files. It can store data files, but using ordinary Git for large, frequently changing training datasets often makes an ML project slow and difficult to maintain. A practical example Imagine a food delivery team training a model using a large file called orders.csv. On Monday, the team saves the dataset in Git. On Tuesday, thousands of new orders arrive, so someone replaces the file and commits it again. They repeat this every day. Even if the project folder contains only the latest orders.csv, Git keeps information about earlier versions in its history . Git can compress and reuse some content, but repeated changes to large datasets may still make the repository grow substantially. Now a new engineer wants to work on a tiny change to the model's Python code. Cloning the repository may require transferring a large amount of historical data that has nothing to do with that code change. What problems does this cause? 1. Growing repository history: Removing a large dataset from the latest commit does not automatically erase its older versions. The history may remain large until it is deliberately rewritten. 2. Slower collaboration: Large repositories can take longer to clone, fetch, push, and process in continuous integration jobs. That affects teammates who only need the code. 3. Difficult reviews: Git is good at showing which lines of source code changed. Large binary or compressed datasets often cannot be reviewed as a meaningful line by line difference. Even large text datasets can produce noisy changes. 4. Storage and hosting limits: Git hosting providers may restrict file sizes or repository activity. For example, GitHub warns when a normal Git file exceeds 50 MiB and blocks files over 100 MiB; these are GitHub rules , not universal limits of Git. 5. Data exposure risk: Training datasets may contain customer details. If private data is committed, deleting the file from the current branch is not enough to remove it from earlier history or existing copies. What is a better approach? Keep training code and small version references in Git , while storing the actual large dataset in an appropriate external location. For the food delivery project, the arrangement could be: Git repository: Training scripts, tests, configuration, and a reference identifying the exact dataset version. Data storage: The full order dataset, protected with appropriate permissions. Training job: Reads the chosen dataset version and records which code commit and data version it used. Two common ways to manage the connection are: Git LFS: Stores a small pointer file in Git and keeps large file content in separate LFS storage. This can work for large files that need to follow Git versions, subject to storage and bandwidth limits. DVC (Data Version Control): Tracks dataset and pipeline metadata that can be committed to Git, while data lives in a cache or external storage such as object storage. This is often useful when datasets change regularly during ML experiments. Neither tool makes private data safe automatically; storage permissions and access controls still matter. Does this mean Git should never contain data? No. A tiny, non sensitive sample dataset used to demonstrate a feature or run a test is often fine to commit. The concern is large, frequently updated, or sensitive data—not every CSV file. In short: Ordinary Git is excellent for tracking the code that builds a model, but bulky training data history can slow down everyone using the repository. Store large datasets separately and keep enough version information to reproduce the training run.

How do Git tags map to ML releases?

A Git tag is a readable name for a specific point in a repository's history. In an ML project, teams often use tags such as ml pipeline v1.2.0 to identify which version of the code was used for a release. But there is an important distinction: a Git tag identifies code; it does not identify a complete trained model by itself. A model also depends on data, training settings, and the resulting model artifact. A practical example Imagine a food delivery company with a model that predicts delivery times. The team improves the feature processing code and finishes its review and tests. It creates the Git tag ml pipeline v1.2.0 for the approved code. That tag points to the exact commit used for this code release. A training job then uses: Code: ml pipeline v1.2.0 (and its Git commit). Data: A particular snapshot of historical delivery orders. Configuration: Recorded features, training settings, and dependency versions. Result: A trained artifact registered as, for example, delivery time model version 23. A release record links these pieces. If the team later asks, “What produced model version 23?”, they can identify the exact code and training inputs. How a team creates and uses a release tag 1. Review and test the code. Required tests and approvals should pass before the team identifies an approved code version. 2. Create an annotated tag. Annotated tags store a message and information about who created the tag. For example: 3. Build from the tagged commit. The CI/CD process can use that exact revision when building or releasing the training pipeline or inference service. 4. Record the model lineage. When training creates a model, save the code commit, dataset version, configuration, training run ID, and registered model version together. 5. Approve the actual model deployment separately. A code release or Git tag does not guarantee the resulting model is accurate or safe to deploy. GitHub Releases can add release notes and downloadable software assets around a Git tag, but a model registry is usually the better place to manage trained model versions and their deployment status. Can one Git tag produce several models? Yes. The same tagged training code might run once using September data and again using October data. Those runs can produce different model versions, even though the Git tag has not changed. The opposite is also possible: the same trained model artifact may be served by different versions of application code. That is why code versions, model versions, and deployment releases should be recorded as separate identifiers . What should teams avoid? Do not move or reuse an existing release tag casually. Treat released tags as stable references and protect them according to your repository policy. Do not store large trained model files in ordinary Git just because they belong to a release. Keep them in an artifact store or model registry. Do not treat a tag as proof that a model was tested or deployed. Record the evaluation outcome, approval, and actual deployment version. Do not assume reverting code restores an earlier model. A rollback needs the correct model artifact and compatible serving code or configuration. In short: Use Git tags to name trusted code versions , model registries to identify trained model versions , and release records to connect code, data, evaluation, and deployment.

What is Infrastructure as Code (IaC) in MLOps?

Infrastructure as Code (IaC) means describing and managing computing infrastructure in version controlled configuration files instead of setting up every resource by clicking through a cloud console. In MLOps, that infrastructure might include the storage for training data, computing resources for model training, networking, access permissions, or a service that hosts predictions. The key idea: We can review, repeat, and change the environment around a model as systematically as we change its code. A beginner friendly example Imagine a food delivery company building a model that predicts how long an order will take. A developer sets up the first ML environment manually: A cloud storage bucket holds training data. A compute service runs model training jobs. A prediction endpoint serves the model to the app. Access rules decide who can use these resources. The system works, but the company now needs a separate test environment. Someone must remember every setting and recreate it manually. A missed permission, wrong region, or different machine size can cause a failure or unexpected cost. With IaC, the team describes the required resources in configuration files. An infrastructure tool creates or updates the environment from those definitions. The same approach can help create a similar test environment without rebuilding everything from memory. How does IaC work? A typical workflow is: 1. Define: Write configuration describing the desired infrastructure—for example, storage, training compute, and permitted network access. 2. Review: Store the configuration in Git and review changes through a pull request. 3. Preview: Use the IaC tool to see the proposed infrastructure changes before making them. 4. Apply: After approval, let the tool create or update the resources. 5. Maintain: Update the configuration when requirements change, and periodically check for differences between configured and actual infrastructure. Define → Review → Preview → Apply → Maintain For example, when a training workload needs more compute, the team can propose a configuration change instead of silently modifying a server by hand. Common IaC tools Terraform: Defines resources using declarative configuration and supports many cloud platforms through providers. A common workflow uses terraform plan to review changes and terraform apply to make them. AWS CloudFormation: Manages AWS infrastructure from templates. Azure Bicep: Defines and deploys Azure resources. These tools are alternatives rather than ingredients that every MLOps stack must use. Why is IaC useful in MLOps? Consistency: Training and test environments can follow the same reviewed definitions, with explicit differences when needed. Traceability: The team can see when an infrastructure setting changed and who reviewed it. Repeatability: It is easier to rebuild approved resources after mistakes or create a new environment. Safer changes: A preview and approval step can catch unexpected replacements or deletions. Collaboration: ML engineers and platform engineers work from shared configuration rather than undocumented manual steps. Important limitations IaC does not train models or guarantee identical predictions. Reproducible ML still needs versioned code, data, dependencies, and training settings. IaC handles infrastructure; ML pipelines handle steps such as data validation, training, and evaluation. IaC also needs safeguards. For example, Terraform state can contain sensitive information , so teams should store state securely, restrict access, and use suitable locking for shared workflows. They should also review plans for potentially destructive changes and avoid putting secrets directly in configuration files. In short: IaC makes ML infrastructure repeatable, reviewable, and easier to maintain , reducing configuration mistakes as models move from experiments to production.

How do declarative and imperative infrastructure management differ?

Declarative infrastructure management describes what the infrastructure should look like . Imperative infrastructure management describes the steps to make it look that way . Both approaches can automate MLOps infrastructure. The main difference is who decides which actions to perform : the infrastructure tool or the person writing the script. A practical example Imagine a food delivery company needs cloud storage for trained models and a compute service for running delivery time predictions. Imperative approach: describe the steps An engineer writes a script that says: 1. Create a storage bucket for model artifacts. 2. Turn on bucket versioning. 3. Give the prediction service permission to read approved artifacts. 4. Create a compute resource for serving predictions. 5. Start the prediction service. The script explicitly controls the sequence of operations. It can be convenient for custom procedures, but the engineer must also consider what happens when the script runs again . For example, if the bucket already exists, blindly trying to create it again may cause an error. Well designed imperative scripts can check the current state and safely handle repeated runs. Repetition problems are not inevitable; the logic is simply the script author's responsibility. Declarative approach: describe the desired result Instead, the engineer writes configuration saying the environment should have: One versioned model artifact bucket. A prediction service with the specified compute capacity. Access rules that permit only the intended service to read artifacts. Tools such as Terraform or AWS CloudFormation interpret those declarations and work out the necessary operations. A Terraform workflow typically compares the declared configuration with managed infrastructure, presents a plan, and then applies approved changes. The engineer focuses on the intended end state rather than writing every API call. What happens when requirements change? Suppose the prediction service now needs greater compute capacity. Imperative: The engineer modifies the script to find the existing resource and run the appropriate resize or replacement commands. Declarative: The engineer changes the desired capacity in configuration. The tool computes what it needs to update or replace to match the new declaration. Neither approach guarantees a disruption free update. Some infrastructure changes require replacement or can delete resources, so review the proposed changes before applying them. Which approach should an MLOps team prefer? Declarative management is often a good default for long lived infrastructure: storage, compute, networking, access policies, and the environments used for model training and serving. Imperative steps are useful for ordered operational actions, custom setup tasks, and automation that does not fit neatly into a resource declaration. A combination is common. For example, Terraform may provision a training environment, while a separate job runs the training command after the environment is ready. A few cautions 1. Declarative does not mean automatic safety. Incorrect desired state configuration can still create expensive resources or destroy important ones. 2. The tool's state matters. Terraform tracks managed resources using state. Teams should protect shared state and avoid unreviewed manual changes that create configuration drift. 3. Imperative does not mean non repeatable. Scripts can be written to check existing conditions and behave consistently. 4. Infrastructure is not the ML pipeline. Declaring servers and storage does not by itself validate data, train models, or approve deployments. In short: Declarative = specify the desired infrastructure. Imperative = specify the actions. For MLOps, choose the approach that makes resources easy to review, maintain, and reproduce, using both when appropriate.