MLOps

MLOps applies software engineering and operations discipline to machine learning. It covers reproducible training, model registries, deployment, monitoring, governance, and incident response.

What are common causes of development-to-production failures in ML systems?

A model can score well in a notebook and still fail in production. Development to production failure means that a model or its surrounding system does not behave as expected after deployment—even when offline evaluation looked promising. Two kinds of failure matter: Operational failure: The service does not start, times out, crashes, or cannot reach a dependency. Prediction failure: The service responds normally, but its predictions are wrong, inconsistent, or no longer useful. A beginner friendly example Imagine a food delivery company predicting how many minutes an order will take to arrive. During development, the model achieves a low prediction error. After deployment, customers receive unrealistic delivery estimates. An investigation finds that training used restaurant to customer distance in kilometers , while one production service sends meters . The API does not crash: it returns numbers, but they are unreliable. That is a silent production failure, which can be harder to notice than an error message. Common causes and what they look like 1. Training serving data mismatch Training and inference receive features with different units, names, missing value handling, or preprocessing. For example, a feature called distance km receives the value 3500 when the real distance is 3.5 km. Check: Validate feature schemas and units, and run the same representative records through both preprocessing paths. This is one form of training serving skew . 2. Data leakage or unrealistic offline evaluation A model may learn from information unavailable at prediction time. For example, training uses the actual delivery completion time to predict a delivery delay, even though that value does not exist when the customer places an order. Check: Use time aware validation where appropriate and confirm every feature would have been available at the actual prediction moment. Strong test scores cannot rescue leaked inputs. 3. Model and runtime incompatibility The saved model expects a library, Python runtime, device, or input format that the serving application does not support. The result may be a model loading error or incorrect preprocessing. Check: Load the exact release candidate in a production like serving environment and exercise the real prediction API before rollout. 4. Missing permissions or upstream dependencies A production worker cannot fetch its model from object storage, or an upstream feature service has a timeout. A notebook may work because the developer's personal account has broader access. Check: Test with the application's actual service identity and verify its dependencies, timeouts, and failure handling. 5. Performance assumptions break under real traffic A model may predict quickly for one request but fail latency targets at peak traffic. Common causes include model loading overhead, GPU memory limits, inefficient batching, and downstream calls. Check: Test representative request sizes and concurrency while measuring latency, throughput, memory, and failure rates. 6. Real world data and behavior change After launch, a change in weather, restaurant operations, or customer behavior may make the original training distribution less representative. This is data drift or, if relationships change, concept drift . Check: Monitor input distributions and prediction quality against a meaningful baseline. Retraining is useful only after investigating the cause and validating the candidate replacement. 7. Missing observability or a recovery plan The service may appear healthy because it returns HTTP 200, while predictions become worse. Without model version records, feature statistics, alerts, and a known rollback target, finding the problem takes much longer. Check: Monitor both system health and model quality , record the deployed model version, and rehearse rollback or a safe fallback. A practical pre release checklist Before promoting a delivery time model: 1. Validate the data contract: Expected feature names, units, null handling, ranges, and features available at prediction time. 2. Test the actual release: Start the approved model artifact in the intended serving runtime and send realistic requests. 3. Test failure and load conditions: Missing features, unavailable dependencies, concurrent requests, and a controlled restart. 4. Compare model quality: Check the candidate against a previous model and a stable evaluation threshold, not just its training score. 5. Roll out gradually: Send a limited portion of traffic to the new model when possible, watch system and prediction metrics, and keep a rollback path. How is this different from environment parity? Environment parity, covered earlier, addresses one root cause: unexpected differences between development and production . This lesson takes the wider view of why an ML system fails at launch or afterward —including data leakage, live traffic, unreliable upstream systems, and changes in real world behavior. In short: A good offline metric proves only part of readiness. Production success requires correct inputs, compatible software, dependable infrastructure, meaningful monitoring, and safe release controls.

How do you define ownership boundaries among ML platform components?

An ML platform contains several connected components: data pipelines, feature processing, training workflows, model registries, prediction services, and monitoring systems. An ownership boundary explains which team is accountable for each component , what that team promises to others, and where responsibility passes to another team. This is different from simply listing job titles. Even if everyone understands what an ML engineer does, a production incident can still be delayed if nobody knows who owns the feature service or the model serving API . A beginner friendly example Imagine a food delivery company with a model that predicts when an order will arrive. Its ML platform has a shared data pipeline, a feature store, a training system, a model registry, and a prediction API. One morning, predictions become inaccurate because restaurant preparation time features are several hours old. Without clear ownership, the application, data, and MLOps teams may all assume that someone else will investigate. With clear boundaries: The data pipeline owner checks whether new restaurant events arrived on time. The feature pipeline owner checks whether the right feature values were computed and published. The ML platform owner checks whether the shared feature store service is healthy. The prediction service owner checks model requests and the impact on customers. One designated incident coordinator follows the issue across teams. Ownership should make cooperation faster, not encourage teams to say, “That is not my problem.” How can you assign ownership by component? For each component, identify one accountable owning team , even when several teams contribute. 1. Data ingestion and data quality A data engineering team might own the event ingestion pipeline, data contracts, delivery schedules, and data freshness. It should publish clear expectations such as, “New order events are available within five minutes,” alongside a support contact. 2. Feature definitions and feature infrastructure The model or applied ML team can own the meaning of features—such as how restaurant preparation time is calculated. A platform team can separately own the shared feature store service : its uptime, storage, API, and access controls. This is an important boundary: owning a feature's business logic is not the same as owning the infrastructure that serves it . 3. Training and orchestration The ML platform team can own shared schedulers, compute templates, and workflow tooling. The team developing the model can own training code, evaluation checks, and the correctness of its pipeline. 4. Model registry and release approval The platform team can maintain the registry's availability and versioning mechanism. The model owning team should define acceptable model quality and provide required release evidence. Assign an explicit approver for promotion into production; platform access alone should not imply permission to approve any model. 5. Prediction API and runtime An application or ML engineering team can own the API contract, prediction behavior, and customer facing performance objectives. A platform team might own the underlying serving infrastructure. Both responsibilities must be documented, particularly when an incident spans them. 6. Monitoring and security A platform or reliability team may operate shared logging, alerting, and access management tools. The model owning team should set and investigate model specific quality or drift thresholds. Security specialists establish policy and review sensitive controls; operating teams implement those controls in their components. These are example assignments , not a universal organization chart. Smaller teams may own several components. The critical requirement is that the boundaries remain explicit. What should every component's ownership record contain? A short entry in a service catalog or repository can include: Component and purpose: What it provides, and to whom. Accountable team: One team responsible for maintaining it and arranging support. Interface or contract: Inputs, outputs, schema, API, freshness expectations, and important service objectives. Change authority: Who can modify it, review breaking changes, and approve a release. Incident route: Alert destination, runbook, escalation contact, and fallback. Dependencies: Upstream providers and downstream consumers. For example, Backstage supports a software catalog with an explicit component owner and recorded relationships to APIs and other components. A simple shared document can serve the same purpose for a small team. How do you resolve ownership conflicts? Use a lightweight RACI style decision for high risk operations: Responsible: Who carries out the work? Accountable: Who makes the final decision and owns the outcome? Consulted: Who provides input before the change? Informed: Who must hear about the result? For example, the ML team may implement a model update, a designated model owner may approve its quality, and the platform team may execute the deployment. Document the actual decision path rather than assuming an individual's job title grants approval. For cross component incidents, name a coordinating responder and let each component owner investigate its own part. Create an escalation path for gaps where no owner exists. What is the common mistake? Assigning a team to a tool without defining responsibility for the service delivered by that tool . “The platform team owns the feature store” does not answer who is accountable when a feature's value is wrong, even though the feature store API is healthy. In short: Give each ML platform component an accountable team, a clear interface, a change owner, and an incident route. Define shared boundaries explicitly so problems can be diagnosed and releases approved without confusion.

What are control plane and data plane responsibilities in an ML platform?

An ML platform has to manage workloads and run workloads . Separating these activities into a control plane and a data plane makes that distinction clear. Control plane: Decides what should run, where, with which configuration, and how its lifecycle is managed . Data plane: Performs the actual work—such as training a model, processing features, or serving predictions . A useful way to remember it is: the control plane directs the work; the data plane executes it . The precise components depend on the platform. A beginner friendly example Imagine a food delivery company using a model to estimate when each order will arrive. An engineer wants to release delivery time model version 14 . Control plane steps: 1. The deployment API accepts an authorized request to deploy the approved model version. 2. The platform records the desired deployment settings, including the model artifact, number of serving replicas, and compute requirements. 3. A scheduler or controller arranges for serving instances to run and checks whether they reach the desired state. 4. If traffic grows, the platform can change the desired number of replicas according to its scaling policy. Data plane steps: 1. A customer places an order and the application sends a prediction request to the running inference endpoint. 2. A serving instance loads or uses the deployed model, processes the required features, and calculates a delivery time estimate. 3. The inference endpoint returns the estimate to the application. The deployment request belongs to the control plane; the prediction request belongs to the data plane. An ordinary customer prediction should not have permission to change the deployment. What typically belongs to each plane? Control plane responsibilities Lifecycle management: Create, configure, update, scale, suspend, or remove training jobs and inference deployments. Desired state management: Track which model version and runtime configuration should be active, then reconcile the actual system toward that state. Scheduling and orchestration: Allocate jobs or serving workloads to appropriate CPU/GPU resources and coordinate rollouts. Management authorization and policy: Check who is permitted to launch training jobs, update endpoints, or change resource limits. Management visibility: Show job status, deployment readiness, and infrastructure health to operators. Data plane responsibilities Training execution: Read authorized training data, run feature transformations and optimization steps, and write checkpoint or model artifacts. Online inference: Receive and validate prediction inputs, execute preprocessing and model computation, and return responses. Batch processing: Run scheduled scoring or feature processing work over larger datasets. Runtime performance: Handle request throughput, batching, latency, errors, and workload specific resource use. Data access enforcement: Use workload identities and controls to access only the datasets, models, and services needed for execution. Some supporting systems—such as model registries, object storage, metrics services, and gateways—support both planes or sit beside them. A platform architecture should describe their interfaces rather than force every component into exactly one box. A real world implementation: Kubernetes and KServe In Kubernetes , the API server, scheduler, and controllers are control plane components. Worker nodes run the containers that carry out actual workloads. For model serving, KServe applies a similar pattern: Its control plane manages the lifecycle of inference services, creates or updates underlying Kubernetes resources, and coordinates scaling. Its data plane consists of serving runtimes that handle incoming prediction or generation requests. In Azure Machine Learning , the distinction also appears at the API level: creating or updating an online endpoint is a control plane operation , while sending an input for scoring is a data plane operation . Why does the separation matter? 1. Safer access: An application may need permission to call a model without having permission to replace it. Separate management and inference permissions wherever the platform supports them. 2. Independent performance goals: Control plane APIs emphasize correct state changes and operational reliability. Online inference needs low latency, throughput, and predictable request handling. 3. Easier fault diagnosis: If the deployment API or scheduler is unavailable, existing healthy prediction servers may still answer requests, although deployments and scaling can be affected. If an inference worker fails, predictions may fail even while management APIs work. Actual behavior depends on the platform's dependencies and redundancy. 4. Easier scaling: Management services and GPU heavy inference workers can use different resource pools. A new model deployment does not require routing every inference request through the management API. A common misunderstanding The data plane is not simply “the database” and the control plane is not necessarily a separate physical cluster. These names describe responsibilities and traffic paths , not fixed machines. The same product may expose both kinds of API, and different platforms may draw the boundary differently. In short: The control plane configures and supervises ML workloads ; the data plane executes the training, processing, or inference . Keeping these responsibilities distinct helps secure deployments, scale efficiently, and troubleshoot failures.

How should an ML team choose between a monorepo and multiple repositories?

A monorepo keeps multiple related projects in one Git repository. A multirepo approach gives those projects separate repositories. For an ML team, the choice is less about folder names and more about how often components change together, who maintains them, and how independently they must be released . Neither approach is universally better. A beginner friendly example Imagine a food delivery company with three pieces of software: A feature library that calculates distance, restaurant preparation time, and time of day features. A training pipeline that uses those features to train delivery time models. A prediction API that uses the approved model to estimate arrival time. A change to the distance calculation may require coordinated updates to both training and inference. If one small team maintains all three, a monorepo can make that change easier to review in one pull request . Later, a dedicated platform team manages a shared feature service for many products, while another team releases the customer facing API several times a day. Separate repositories may give those teams clearer access controls and independent workflows. When is a monorepo a good fit? Choose a monorepo when several components are tightly related and frequently need changes together. Benefits include: 1. Atomic cross component changes: Update the shared feature logic, training tests, and inference tests in one reviewed change. 2. Shared standards: Maintain common linting, dependency policies, templates, and CI conventions. 3. Discoverability: Engineers can inspect callers and dependent code without searching across many repositories. 4. Easier coordinated refactoring: A feature schema change can include updates to its consumers in the same commit. Challenges include: CI cost: Testing and rebuilding every project after every small change becomes expensive. Use dependency aware or path aware pipelines to run the relevant checks; include downstream tests when shared components change. Ownership: A shared repository does not mean everyone should approve changes everywhere. CODEOWNERS and protected branches can require reviews from relevant teams. Access isolation: Repository wide access may not fit projects with stricter confidentiality or separate compliance requirements. Repository size: Keep large datasets, experiment outputs, and model weights in appropriate data or artifact stores, not ordinary Git history. Important: A monorepo does not force every ML service to deploy at once. Training and prediction services can have separate build pipelines, versioned artifacts, and release schedules. When are multiple repositories a better fit? Choose separate repositories when components have clear ownership boundaries, limited shared changes, or different access and release requirements . Benefits include: 1. Independent releases: A prediction API can ship without merging changes to unrelated training tools. 2. Repository level permissions: Teams can more easily limit access to sensitive or regulated codebases. 3. Team autonomy: Projects can choose different tools or workflows when there is a genuine reason. 4. Smaller local scope: Each repository has its own review queue and CI setup. The trade off is cross repository coordination . If a shared Python feature package changes, maintainers may need to publish a new version, update consumers, and verify compatibility in more than one repository. Without automation, dependency versions and standards can drift. A practical decision checklist Ask five questions before splitting or combining repositories: How often do changes span components? Frequent coordinated changes favor a monorepo. Can components release independently? Separate repositories can simplify independent governance, although a monorepo can also support independent deployments. Who must access or approve each codebase? Strong repository level isolation favors multiple repositories. How much code is shared? Extensive shared code favors a monorepo unless stable, versioned packages already provide a good boundary. Can CI scale? A growing monorepo benefits from affected project tests, build caching, and clearly defined dependency relationships. A small team's sensible starting point might be one repository with separate features/, training/, and serving/ projects. As responsibilities change, an independent platform component can be extracted if the benefits outweigh package publishing and integration overhead . Two mistakes to avoid Mistake 1: One repository means one giant application. Repository boundaries and deployable service boundaries are different. A monorepo can contain multiple independently deployable services. Mistake 2: Every ML model needs its own repository. Splitting dozens of nearly identical pipelines can multiply CI files, dependency upgrades, and shared library releases without improving ownership. The earlier question on ML project organization explained where notebooks, reusable packages, tests, and pipelines belong. This question is about choosing the collaboration and release boundary across projects. Large datasets and trained models still belong in versioned storage systems, regardless of repository strategy. In short: Prefer a monorepo when ML components change together and benefit from shared tooling. Prefer multiple repositories when ownership, access, or releases need stronger independence. Revisit the choice as the team grows rather than treating it as permanent.

What is a software bill of materials for an ML service?

A Software Bill of Materials (SBOM) is a structured inventory of the software components used to build an application—similar to an ingredients label on packaged food. For an ML service, it helps engineers answer: Which Python libraries, operating system packages, and other software dependencies are inside the version we deployed? This matters because an ML model may work correctly while the software around it contains an outdated library, a problematic dependency, or a license the team needs to review. A beginner friendly example Imagine a food delivery company running an API that predicts how long an order will take. The service uses a trained model plus Python, numpy, scikit learn, a web framework, and packages inherited from its container's base image. One day, the security team learns that a particular software library version may be vulnerable. Instead of asking every engineer to guess which projects use that library, they inspect the SBOM associated with the deployed prediction service image . The inventory can show whether that software component was discovered in the image and which version is present. This helps the team decide which releases need investigation. What information can an SBOM contain? Depending on the standard and generation tool, an SBOM can record: Component name and version: For example, the exact installed version of a Python dependency. Supplier or package identifiers: Information to distinguish similarly named packages. Dependency relationships: Which components depend on others. License information: Useful for reviewing software usage obligations. File or package hashes: When available, identifiers that help confirm the recorded artifact. SBOM metadata: Who or what generated it, when, and which software artifact was examined. Two widely used machine readable SBOM formats are SPDX and CycloneDX . An SBOM is an inventory, not a report that every component is safe . Vulnerability tools can analyze the inventory against advisory data; assessing and fixing findings is a separate activity. How can you generate one? A team can generate an SBOM from a built container image , not just its source code dependency declarations. This can capture software supplied by the base image as well as installed application packages. For example, after building an image called delivery time api:1.0, a tool such as Syft can scan it: The command writes a machine readable CycloneDX JSON inventory to sbom.json. It assumes Syft is installed and that the named image is accessible. Actual discovered components depend on the image contents and scanner coverage. A common release workflow is: 1. Build the approved ML service image. 2. Generate its SBOM automatically in CI/CD. 3. Associate the SBOM with the exact image digest and release record. 4. Review newly discovered packages and relevant security or license findings. 5. Retain the SBOM with the release, and regenerate it when the image is rebuilt. This is particularly useful when one organization has many training, batch scoring, and inference images, each with a different software inventory. Is an SBOM the same as a model card or dataset record? No. A conventional software SBOM primarily describes software composition . It does not automatically tell you: Which examples or datasets trained a model. Whether its predictions are accurate or fair. Which evaluation results justified approval. Whether a model file was replaced after approval. Use model registries, model cards, dataset lineage, and artifact versioning for those questions. Some emerging AI specific bills of materials cover additional AI assets, but a normal package scanning SBOM should not be assumed to include them. How does this connect to earlier MLOps concepts? A dependency lockfile records a resolved set of packages the project intends to install . An SBOM inventories components found in a particular built or deployed artifact . They complement one another, but they are not interchangeable. Likewise, a fixed container image digest helps tie the SBOM to a precise released artifact. If the image changes, the relevant SBOM should be updated. In short: An SBOM is the software ingredients list for an ML service. Generate it from the release artifact, retain it with the release, and use it to understand dependencies and support software supply chain reviews.

Why should dependency vulnerabilities be scanned in an ML delivery process?

Dependency vulnerability scanning checks the software libraries and packages used by an ML system against databases of known security flaws . An ML model can predict accurately while the API, training worker, or container running it contains a vulnerable package. That makes scanning part of releasing and maintaining a safe ML service , not just an optional security audit. A beginner friendly example Imagine a food delivery company deploying an API that predicts delivery times. Its service uses Python, a web framework, numerical libraries, and a container base image. Before a release, an automated scanner reports that one installed package version has a published security advisory. This is a hypothetical finding , not a claim about a specific library. The team checks whether the vulnerable functionality can actually be reached in its deployment, whether a patched version exists, and what the business impact could be. If an upgrade is appropriate, it updates the dependency, reruns model serving tests, rebuilds the image, and scans again. Without scanning, the model's accuracy tests might all pass while the vulnerable software remains deployed. What should an ML team scan? 1. Direct Python dependencies: Libraries explicitly used by training, batch scoring, and inference code. 2. Transitive dependencies: Packages installed because other packages require them; these may be overlooked by developers. 3. Container and operating system packages: System libraries and utilities inherited from the base image, not only what appears in requirements.txt. 4. Deployed artifacts: The actual container image or environment released to production, because its contents can differ from the source dependency files. For GPU workloads, also consider the security and compatibility of relevant accelerator runtimes and drivers. A Python only scan cannot evaluate all host or container components. A simple scanning example A team can use pip audit to check the dependencies in a Python requirements file: After building a prediction service image, it can use Trivy to check detected language packages and supported operating system packages: These commands assume the tools are installed, the requirements file is available, and the image can be accessed. They report known vulnerabilities according to the tool's data sources and detection capabilities; a clean report is not proof that the service is secure . Where should scanning fit in delivery? On pull requests: Detect newly introduced vulnerable dependencies before changes merge. During the build: Scan the resolved dependencies and built container image before release. At release approval: Review findings against the team's documented risk policy; block a release when an unacceptable risk remains. After deployment: Rescan or continuously monitor released artifacts. A vulnerability may be disclosed after an unchanged image goes live. A practical pipeline is: Dependency change → Build → Scan → Assess findings → Fix or approve a time limited exception → Test → Release → Rescan The scan should produce a visible report and a clear person or team responsible for unresolved findings. How should vulnerabilities be prioritized? Not every alert has the same operational risk. Consider: Severity: How serious is the reported flaw according to relevant advisories? Exposure: Can an attacker reach the affected functionality in this particular training job or prediction service? Evidence of exploitation: Is the vulnerability known to be exploited, for example through CISA's Known Exploited Vulnerabilities catalog? Available remediation: Is there a patched release, a replacement package, or a compensating control? Business impact: Is this a public prediction API, an internal isolated batch job, or a component with access to sensitive data? For example, a remotely exploitable flaw in an internet facing inference API generally deserves more urgent attention than an unreachable issue in an isolated development only tool. Document decisions; do not silently ignore alerts or automatically upgrade every ML library without compatibility testing. What does scanning not replace? An SBOM inventories components. Scanning identifies known security advisories affecting detected components . They complement one another but are not the same process. Dependency pins and lockfiles preserve chosen versions; they do not guarantee those versions remain safe forever. Application and model security testing still matter. A dependency scanner does not necessarily detect custom code bugs, malicious model files, exposed credentials, or weaknesses in ML specific workflows. In short: Scan dependencies before release and throughout their deployed lifetime . Investigate findings in context, update safely, and verify fixes without breaking model behavior.

What is the shared responsibility model for cloud-hosted ML systems?

The shared responsibility model explains which security and operational tasks belong to a cloud provider and which remain with the team using the cloud . The basic idea is: Provider: Secures the underlying infrastructure and the parts of the cloud service it operates. Customer (your ML team): Secures its data, access decisions, model/application code, and the cloud settings under its control. The exact boundary depends on the service . Renting virtual machines gives your team more work than using a managed ML platform. A beginner friendly example Imagine a food delivery company training a model to predict delivery times and serving predictions through a cloud endpoint. The team uses a managed platform such as Amazon SageMaker AI . The cloud provider is responsible for protecting its physical datacenters, physical machines, and underlying managed service infrastructure. The food delivery team must still decide: Who can read order histories and customer related data. Which training job or prediction endpoint may access each dataset. Whether the model and application handle inputs safely. Which network settings, encryption options, retention policies, and logs are required. Whether the model is accurate enough and meets the company's privacy and compliance requirements. The provider running the service does not automatically make an overly permissive data bucket, an exposed inference endpoint, or an unreliable model safe. How responsibilities change with service type 1. Self managed virtual machines (IaaS) If the team trains and serves models on cloud virtual machines, the provider secures the underlying cloud hardware and virtualization infrastructure. The team usually handles the guest operating system, relevant patches, ML runtime and container stack, application, access rules, data, and model security. For example, if an engineer runs a prediction API on an EC2 instance, securing the instance's guest OS and application is generally the customer's job. 2. Managed ML services (PaaS like) If the team uses managed training jobs and inference endpoints, the provider operates more of the platform: service infrastructure, scheduling machinery, and parts of the managed runtime depending on the offering. The customer still controls its own ML application or supplied containers, data permissions, configured network access, and model quality. Do not assume all runtime patching is handled by the provider ; consult the specific service and image documentation. 3. Fully hosted AI applications (SaaS like) When a provider supplies a complete application, it manages more of the software stack. The customer still needs to govern users, uploaded data, permitted integrations, settings, and appropriate use of outputs. The names IaaS, PaaS, and SaaS are useful guides, but read the security documentation for the exact service rather than guessing from its category. Where do ML specific responsibilities appear? Training data: The provider can offer encryption and storage services, but the ML team chooses what data is uploaded, who can read it, and how long it is retained. Identity and permissions: The platform offers IAM and related controls. The customer designs and applies least privilege policies for people, pipelines, training jobs, and inference services. Models and application code: The customer validates its supplied models, preprocessing, dependencies, inputs, and prediction behavior. Managed infrastructure does not certify model correctness. Network exposure: The provider operates underlying networking; the customer configures permitted endpoint access, private connectivity where needed, and applicable firewall or service controls. Monitoring and incidents: The provider operates its platform and supplies monitoring capabilities. The team must enable suitable logs and alerts, investigate application or model issues, and coordinate with the provider when needed. These are shared workflows , not excuses to leave a task without an owner. A practical ownership checklist Before deploying the delivery time model: 1. List every cloud service the workflow depends on: storage, training, model registry, inference, and monitoring. 2. Find the documented security boundary for each service, including who patches the runtime and who configures encryption, networking, and access. 3. Assign a customer side owner for datasets, permissions, endpoint configuration, model approvals, and monitoring. 4. Test the controls: Use the real workload identity to verify authorized access, and confirm that an unauthorized identity is denied. 5. Review regularly: Changing from self managed containers to a managed endpoint can change who maintains the platform without changing who is accountable for the model's safety and quality. What is often misunderstood? “Our model runs in the cloud, so security is the provider's responsibility.” That is false. Moving to a managed service can reduce the infrastructure you operate, but does not transfer responsibility for protecting your data, choosing safe permissions, or validating your predictions. Likewise, shared responsibility is different from the internal team ownership described in the earlier ML platform lesson. One boundary is between provider and customer ; the other is between teams inside the customer organization . In short: The cloud provider secures what it operates; your ML team secures what it builds, uploads, configures, and authorizes. Check the boundary for each service and make the customer side responsibilities explicit.

What is the difference between CPU, GPU, and accelerator scheduling for ML workloads?

Scheduling means deciding which machine should run an ML job based on the resources that job needs and what is currently available. CPU, GPU, and other accelerators have different scheduling rules. A scheduler that only checks free CPU and memory may still be unable to place a job that requires a particular GPU or TPU. 1. CPU scheduling: flexible and shareable CPUs are general purpose processors. They work well for data cleaning, feature processing, orchestration, and many smaller ML models. In Kubernetes, a job can request a fraction of a CPU . For example: cpu: "500m" means a request for 0.5 CPU . cpu: "2" means a request for 2 CPUs . Kubernetes uses CPU requests when choosing a suitable node. CPU limits , if configured, cap the CPU time a container can use. Depending on the node's current workload and limits, multiple CPU based jobs can share available compute. Example: Two preprocessing jobs that each request 500m account for a total request of 1 CPU . 2. GPU scheduling: device availability matters GPUs accelerate parallel numerical work such as deep learning training and some forms of inference. But requesting 1 GPU is different from requesting 1 CPU . Kubernetes typically discovers GPUs through a device plugin , which advertises a schedulable resource such as nvidia.com/gpu. The job asks for a discrete number of GPU resources. A simplified container resource specification is: Here the workload asks for 2 CPUs , 8 GiB of RAM , and 1 GPU . With the standard Kubernetes GPU resource model, GPU limits also establish the GPU request. The GPU plugin, compatible drivers, and suitable GPU nodes must already be configured. GPUs are normally allocated as whole advertised devices , even if the program does not fully use the device's compute power or memory. GPU memory requirements and framework compatibility must also be checked; getting a GPU allocation does not guarantee that a large model will fit in memory. 3. A numerical scheduling example Suppose a cluster has one worker node with: 8 allocatable CPUs 32 GiB allocatable RAM 2 advertised GPUs A training job requests 2 CPUs, 8 GiB RAM, and 1 GPU . Job A: Runs and receives one GPU. Job B: Runs and receives the other GPU. Job C: Also needs one GPU but cannot be placed on this node. After A and B, the node still has 4 CPUs and 16 GiB RAM unrequested , yet Job C remains Pending because no GPU resource is available. A CPU only preprocessing job needing 0.5 CPU and 2 GiB RAM may still fit, subject to other scheduling constraints. This is why monitoring only CPU utilization can be misleading in an ML cluster. 4. Can multiple ML jobs share one GPU? Sometimes, but GPU sharing requires explicit support and configuration . Time slicing: Supported NVIDIA configurations allow several workloads to take turns using one physical GPU. They share the underlying device; time sliced allocations do not provide separate GPU memory or fault isolation . Multi Instance GPU (MIG): On supported NVIDIA GPUs, a physical GPU can be divided into hardware isolated instances with defined resources. The scheduler can allocate the advertised instances to different jobs. These approaches are not identical, and their resource names and capabilities depend on the device plugin and platform. Do not assume that requesting nvidia.com/gpu: 0.5 works : ordinary Kubernetes extended resources are not requested as fractional devices. 5. What about TPUs and other accelerators? A TPU is a specialized accelerator often used for large tensor computations. In supported Google Kubernetes Engine configurations, TPU resources can be exposed as google.com/tpu. TPU placement may require a specific chip count, compatible slice type, and topology . A multi chip training job might need a coordinated set of devices rather than just any single available accelerator. Other accelerator families can have their own device plugins, supported shapes, memory constraints, and scheduling requirements. 6. How should an MLOps team choose a scheduling approach? CPU jobs: Use requests for modest data processing and control tasks. Avoid reserving expensive accelerators when the workload cannot benefit from them. GPU jobs: Request the correct GPU type and count; verify model memory needs, drivers, and scheduling availability. Consider sharing only when isolation and performance requirements allow it. TPU or specialized accelerator jobs: Match framework support, chip topology, and the platform's capacity model. Mixed pipelines: Run CPU heavy ingestion and preprocessing separately from GPU heavy training or inference where practical. Production planning: Monitor job queue time, CPU and accelerator utilization, device memory, throughput, and cost. Node autoscaling or reservations may help when jobs wait for scarce accelerators. Scheduling is not the same as executing kernels on the GPU. The cluster scheduler chooses where a workload runs ; the accelerator runtime schedules the lower level computations once the workload starts. In short: CPUs are generally allocated in flexible, shareable quantities, while GPUs and specialized accelerators require compatible device resources and often have stricter placement constraints. Good MLOps scheduling matches the workload to the right hardware without leaving expensive devices idle or jobs stuck Pending .

How do you estimate compute and storage needs before productionizing a model?

Capacity planning means estimating the CPU, memory, accelerator, and storage needed for an ML system to meet its workload and performance goals without excessive cost. The best starting point is measured behavior on representative data , not choosing a large machine because a model seems complex. What should you estimate first? Separate the system into training , data preparation , and inference , because they have different demands. Training: Dataset size, model size, epochs, batch size, CPU/GPU memory, checkpoint frequency, and acceptable training duration. Data processing: Volume of input records, parallel workers, temporary storage, memory used by transformations, and the time available to finish each batch. Online inference: Peak requests per second (RPS), input sizes, batch behavior, latency target (for example p95), memory per model replica, and uptime requirements. Storage: Raw and processed datasets, model versions, checkpoints, logs, retention periods, replication, and backups. Start with a small benchmark, then extrapolate carefully. Performance does not always scale linearly when concurrency, I/O, or accelerator utilization changes. A beginner friendly numerical example Imagine a food delivery company preparing to launch a model that predicts delivery times. Step 1: Estimate serving capacity The product team expects 60 prediction requests per second at peak and wants p95 response latency below 200 milliseconds . A representative load test shows that one proposed inference replica can handle 15 requests per second while meeting that latency target. To preserve headroom for traffic fluctuations, the team initially plans to use only 60% of this tested throughput : Safe planning capacity per replica = 15 × 0.60 = 9 requests/second. Minimum replicas for peak load = ceil(60 ÷ 9) = 7. For a simple one replica failure allowance , plan 8 replicas : if one fails, 7 remain, providing roughly 7 × 9 = 63 requests/second of planned capacity. These are planning assumptions, not guarantees. Re run load tests with all replicas, realistic networking and downstream calls, and a failure simulation. Autoscaling delay and uneven traffic may require additional headroom. Step 2: Estimate serving memory and CPU During the same test, the team observes approximately: Loaded model: 1.2 GiB . Serving runtime and libraries: 0.8 GiB . Request buffers and working memory: 0.5 GiB . Total measured memory = 1.2 + 0.8 + 0.5 = 2.5 GiB per replica. Adding 25% headroom gives 2.5 × 1.25 = 3.125 GiB. An initial 4 GiB allocation per replica is a reasonable trial configuration, subject to peak memory testing. At 8 replicas , planned serving memory is 8 × 4 = 32 GiB across the replicas, excluding node and platform overhead . If measurements show each replica needs 1 vCPU , the initial CPU request budget is 8 vCPUs . These totals describe requests or allocations, not necessarily actual consumption. If the model needs GPUs, benchmark GPU throughput and VRAM separately; system RAM alone does not prove a GPU model will fit. Resource availability and accelerator placement were covered in the previous scheduling question. Step 3: Estimate storage Suppose the company stores 100,000 new feature records per day , averaging 2 KB per record (decimal units). Daily data = 100,000 × 2 KB = 200 MB. 30 days of raw records = 200 MB × 30 = 6 GB. If the storage design keeps two separately charged copies , that becomes 12 GB . The team also keeps 20 model artifacts , averaging 200 MB each : Model artifacts = 20 × 200 MB = 4 GB. Combined illustrative subtotal = 12 + 4 = 16 GB. That 16 GB is not a complete production storage budget . Add processed datasets, indexes or metadata, logs, temporary training files, checkpoints, recovery copies, and expected growth. Confirm whether a cloud storage service's built in redundancy is already included in its quoted price; do not count the same replication twice. How should training capacity be estimated? Run a representative trial and record: Training examples processed per second and total wall clock time. Peak RAM and GPU VRAM, not only average use. CPU/GPU utilization and time spent waiting for storage or data loading. Model checkpoint sizes and the disk space needed while creating them. For example, if a benchmark handles 200,000 training records in 10 minutes , processing 2 million records might initially suggest 100 minutes at the same throughput. But a larger dataset may expose I/O bottlenecks, larger intermediate data, or different batching behavior. Verify with a longer test before committing to production deadlines. A practical preproduction checklist 1. State the requirements: Peak RPS, p95/p99 latency, batch deadlines, retention, availability, and growth assumptions. 2. Benchmark candidate hardware: Measure resource peaks and realistic throughput with production like inputs. 3. Calculate an initial footprint: Compute replicas, CPU and memory allocation, accelerator requirements, and retained storage. 4. Include operational headroom: Rolling deployments, failures, autoscaling delay, checkpoints, and data growth. 5. Estimate costs: Multiply chosen resource quantities by the provider's current rates and expected runtime, plus storage, network transfer, and supporting services. 6. Validate, then monitor: Stress test the proposed setup; after launch, compare actual latency, queue time, utilization, and storage growth with forecasts. Services such as Amazon SageMaker Inference Recommender can benchmark candidate endpoint configurations for latency, throughput, utilization, and cost. They help validate a choice but do not replace understanding your workload. What is the common mistake? Sizing from average traffic or model file size alone . A 1 GiB model can require substantially more runtime memory, and a service that handles normal traffic may fail during peaks or rolling updates. In short: Estimate capacity from workload volume and measurements; calculate compute, memory, and retained storage with sensible headroom; then validate those estimates through realistic load tests.

How do containers help standardize training and inference environments?

Containers help standardize ML environments by packaging the code, Python runtime, installed libraries, and other user space dependencies needed to run a workload into a tested image. The same image can be run on a developer's machine, in CI, and on compatible cloud infrastructure instead of manually rebuilding its environment each time. This matters because an ML model is more than its saved weights: training and prediction also depend on how inputs are prepared and how the model is loaded . A beginner friendly example Imagine a food delivery company building a model to predict delivery times. During training, a script converts the distance between restaurant and customer from metres to kilometres. For example, 1,200 metres becomes 1.2 kilometres . A developer trains the model in a notebook with one Python and preprocessing library version. The deployed prediction API runs a different library version and accidentally uses a different preprocessing function. The model file may load, but predictions become inconsistent. The team fixes the delivery process by: 1. Putting the shared feature processing code in a tested Python package. 2. Building a training container image with an approved runtime and dependencies. 3. Running training with a versioned dataset, recording the code revision and image digest. 4. Saving the trained model in a model registry or artifact store , with its version and required input schema. 5. Building a serving container image that imports the same shared feature processing package and a compatible model loading runtime. 6. Testing a known order through both paths: the training side and serving side feature calculations must agree that 1,200 metres is 1.2 kilometres. 7. Deploying the tested serving image and the approved model version together. This gives the team a repeatable, inspectable path from training code → model artifact → prediction service . What exactly does the container standardize? A container image can include: The chosen Python version and required user space operating system libraries. ML framework and preprocessing packages, with tested versions. Application code, including common feature transformations and model loading logic. The executable command or entry point for training, batch scoring, or serving. The image is the packaged template; a container is a running instance of it. Build an image once, verify it, and run that tested image in compatible environments. A registry stores the images for later deployment. Prefer a specific image digest when identifying an approved release: unlike a mutable tag, the digest identifies an exact image. Must training and inference use the same image? No. They need compatible behavior and contracts , not necessarily identical containers. Training image: Might include training scripts, experiment tools, GPU libraries, and distributed training support. Inference image: Might contain only the model loader, validated feature processing, API server, and libraries needed for prediction. A company may train on GPUs and serve the resulting model on CPUs. Separate images keep the serving runtime smaller and avoid unnecessary training only dependencies. The important checks are that the deployed model format is supported, feature definitions and input schemas match, and the serving image passes representative prediction tests. Where feasible, both images can reuse a versioned shared preprocessing package . How does this work in an automated ML pipeline? A simple CI/CD and MLOps flow is: 1. Build and test the training image from reviewed code and dependency versions. 2. Run training as a containerized job with externally supplied data and settings. 3. Register the trained model and its relevant metadata after evaluation. 4. Build and test the serving image against that approved model artifact. 5. Promote the serving image digest and model version together to staging, then production. 6. Monitor predictions and roll back to the previously tested image and model combination if necessary. Platforms such as Amazon SageMaker AI use containers for training and model deployment. They support prebuilt framework images as well as customized images, and permit distinct images for training and inference. What containers do not guarantee Identical datasets: Input data, feature definitions, model artifacts, and their versions must be tracked separately. Identical training results: Randomness, parallel computation, numerical libraries, and hardware can affect outcomes, even when the container image is unchanged. Identical hardware: Containers share the host kernel; CPU architecture, available memory, and GPU drivers must still be compatible. Automatic security: Images still need vulnerability scanning, reviewed dependencies, least privilege execution, and external secret handling. The previous lesson on virtual environments versus containers compared two isolation tools. This lesson focuses on how containerized training and inference work together across the model lifecycle . And unlike compute capacity planning, containerization by itself does not decide how many CPUs, GPUs, or replicas are needed. In short: Containers make training and inference environments easier to build, test, share, and release consistently. Use versioned images, shared feature logic, and validated model artifacts so the service performs the same intended transformations after deployment.