MLOps

MLOps applies software engineering and operations discipline to machine learning. It covers reproducible training, model registries, deployment, monitoring, governance, and incident response.

How does a production training pipeline differ from a notebook workflow?

A notebook workflow is designed mainly for interactive exploration: a data scientist inspects data, experiments with features, and runs cells to understand a problem. A production training pipeline is designed to execute an approved training process repeatedly, with explicit inputs, automated checks, traceable outputs, and predictable failure handling. Neither is inherently better at every task. Notebooks are valuable for discovery; production pipelines turn a validated approach into an operational process. Beginner friendly example: delivery time prediction A food delivery company develops a model to predict how long an order will take. During exploration in a notebook: An engineer loads a sample CSV, plots delivery distances, tries feature transformations, and compares two algorithms. Cells are run interactively; temporary variables remain in the kernel. The engineer saves a promising model to delivery model.pkl. A colleague cannot reproduce the result because a feature engineering cell was changed and some cells ran out of order. In the production workflow: A scheduled job selects a versioned snapshot of completed orders. A data quality check verifies the expected schema and labels. The same reviewed preprocessing and training code runs with explicit parameters. The evaluation job compares the candidate with approved quality thresholds. A uniquely identified model artifact and run metadata are stored. Failed steps produce actionable logs; an unapproved model is not automatically deployed. The algorithms may be identical. The difference is how reliably the organization can execute and trust the result . Five important operational differences 1. Execution: interactive state versus explicit entry points In a notebook, a cell may depend on a variable created by another cell hours earlier. That hidden state can make reruns unpredictable. A pipeline runs documented functions or commands with declared inputs and outputs. Each step can be launched in a clean environment. This makes a failed training job diagnosable without reconstructing someone's notebook history. 2. Data: convenient samples versus governed snapshots Exploration often uses a small sample to move quickly. Production training must identify exactly which dataset, time window, and schema it used. A pipeline should reject missing required fields or invalid data before consuming expensive training resources. Holdout strategy and preprocessing fitted on training data must be consistent with the approved evaluation procedure. 3. Artifacts: local files versus versioned lineage A notebook may leave model.pkl on the developer's machine. Production should store artifacts in a shared, durable location and associate the candidate model with its code revision, dataset reference, parameters, metrics, and run ID . The team should be able to answer: Which training execution produced the model now considered for deployment? 4. Reliability: manual recovery versus defined failure behavior A human can inspect and rerun a failed notebook cell. A production job needs timeouts, visible errors, bounded retries for transient failures, and clear rules for stopping on invalid data or unacceptable model quality. Retries must not accidentally overwrite an approved artifact or release a different candidate. 5. Operating model: one experiment versus repeatable releases A notebook is ideal for comparing ideas and examining unexpected patterns. A production pipeline supports team ownership, CI tests, scheduled or event triggered runs, permissions, monitoring, and an auditable handoff to model registration and deployment. Automation does not mean automatic production promotion. A good pipeline can train automatically while requiring a separate approval for release. A small practical conversion Suppose an engineer has a notebook with 12 cells. Six investigate the data, and the other six implement transformations, training, and evaluation. A sensible conversion is not to schedule the notebook exactly as it is. Instead: 1. Keep exploratory plots and analysis in the notebook. 2. Move reusable preprocessing into a tested Python function such as prepare features(data). 3. Move model fitting into train model(train data, params) and evaluation into evaluate model(model, validation data). 4. Add a command line entry point that takes an explicit dataset snapshot, configuration, and output location. 5. Test that the code runs in a fresh process , not only in the original notebook session. 6. Run that code under a scheduler or pipeline service with data checks, stage logging, and artifact tracking. For example, a local command might be: This command is illustrative ; it assumes the project implements those arguments and has authorized access to the storage location. The same entry point can be invoked by a production job with approved inputs. What measurable difference should the team expect? For an illustrative weekly retraining task, assess whether the new workflow can satisfy requirements such as: 10 of 10 clean environment test runs complete or report a specific controlled failure. 100% of newly registered model candidates have a recorded dataset reference, source revision, configuration, and evaluation result. Invalid input schemas stop the job before training . A model that fails quality thresholds is never approved for automatic promotion . A second engineer can rerun the training process without manually executing notebook cells. These are example acceptance criteria , not universal industry benchmarks. They verify repeatability and controls; they do not guarantee identical floating point results across different hardware. Do notebooks have to disappear? No. Teams can use notebooks for research, visualization, error analysis, and interactive experiments. Some notebook files can be parameterized and executed automatically, but they still need explicit inputs, clean state execution, tests, artifact tracking, and operational failure handling. For stable production workflows, separating reusable business logic from the notebook's presentation and interactive state usually makes maintenance easier. Connection to the previous concept Question 50 explained how to arrange automated training stages and dependencies . This lesson explains why turning a successful notebook into a production process requires engineering changes : extracting stateful code, declaring inputs, making results traceable, and supporting unattended operation. Earlier local development concepts addressed testing across environments; this focuses on the interactive versus operational execution model . In short: Use notebooks to discover what model and features work . Use production training pipelines to make the approved process repeatable, testable, observable, and safe to run without a person clicking through cells .

What is a pipeline component and what makes it reusable?

A pipeline component is a self contained unit of an ML workflow that performs one clearly defined task—such as validating data, preparing features, training a model, or calculating evaluation metrics. Think of it as a function designed to run as part of a larger pipeline . It has an interface (inputs and outputs), implementation code, runtime dependencies, and a predictable success or failure result. A reusable component can be called from different pipelines or multiple times with different inputs without rewriting its internal logic . A beginner friendly example A food delivery company needs to check order datasets before both: A weekly model training pipeline uses completed orders. A daily batch scoring pipeline uses recent order features. Both need the same basic validation checks, so the team creates one component called validate orders. Input contract: orders: a dataset artifact containing an orders CSV file. max missing ratio: a numeric parameter, such as 0.02, meaning a maximum of 2% missing values in the selected required field. Output contract: report: an artifact containing a JSON validation report with row count, missing count, and missing ratio. Completion status: The component fails clearly if the dataset violates its declared rule; downstream tasks are blocked by the pipeline's normal dependency handling. For example, if 1,000 orders include 15 missing delivery distances , the missing ratio is 15 ÷ 1000 = 0.015 = 1.5%. With a 2% limit, the check passes . If 35 distances are missing, 35 ÷ 1000 = 3.5%, and the component fails . A practical component in Kubeflow Pipelines Using the Kubeflow Pipelines (KFP) Python SDK, a simplified component can be written like this: What each part means: @dsl.component defines a KFP pipeline component with a declared runtime and dependencies. Input[Dataset] identifies an incoming dataset artifact, not an in memory Python DataFrame shared between jobs. max missing ratio: float is a small scalar input parameter. Output[Artifact] identifies a persisted report artifact for downstream use or inspection. Exceptions signal failure when inputs violate the contract. The example expects a CSV dataset containing distance km . In a real system, validate additional columns, types, units, and values, and use a tested runtime image with approved dependencies. What makes this component reusable? 1. A single responsibility It validates a declared input contract. It does not also download private production data, train a model, send emails, and deploy a service. 2. Explicit, typed interfaces Other pipelines can understand what it accepts and produces. Use small parameters for values like thresholds and artifacts for files such as datasets, models, and evaluation reports. 3. No hidden environment assumptions Avoid hard coded bucket names, absolute laptop paths, private credentials, or references to variables from another notebook cell. Pass required values explicitly or through an authorized runtime configuration. 4. A controlled runtime Declare compatible dependencies and the container environment. For repeatable releases, use tested images, reviewed dependency versions, and appropriate image pinning. 5. Deterministic and testable behavior where feasible For a fixed input dataset and threshold, validation should return the same result. Test the component alone using small fixtures before connecting it to an expensive training job. 6. Meaningful failure behavior Fail on invalid inputs rather than generating an apparently successful but misleading report. Define which errors should stop a pipeline and which transient execution errors may be retried. 7. Versioning and compatibility Give released component implementations identifiable versions. If an output schema or validation policy changes, verify consumers still work or introduce a new compatible version. How do multiple pipelines reuse it? A training pipeline could connect: training dataset → validate orders(max missing ratio=0.02) → train model A batch scoring pipeline could connect: scoring dataset → validate orders(max missing ratio=0.01) → batch predict Both use the same component definition but pass different input artifacts and thresholds. The pipeline orchestrator instantiates a separate task for each invocation. The example's checks must match the use case: if scoring data does not have a delivery distance field, define an appropriate reusable schema contract rather than bypassing validation. Component versus task versus pipeline Component: A reusable definition of work, much like a function. Task: One invocation of that component with specific inputs and runtime settings. Pipeline: A connected workflow of tasks with defined dependencies. The earlier questions explained which stages form an automated training pipeline and how that differs from notebook execution . Here the focus is the design of each reusable building block inside a pipeline. In short: A reusable ML pipeline component performs one clear job, exposes documented inputs and outputs, declares its runtime, fails predictably, and can be tested and invoked from multiple workflows.

How do pipeline inputs, outputs, and artifacts form a dependency graph?

In an ML pipeline, a dependency graph is created when the output of one task becomes an input to another task. These connections tell the orchestrator what data must be available before a downstream task can run . A component defines what work it can do (covered in the previous question). A pipeline wires specific component invocations together to describe how data moves between tasks . Three building blocks Input: A value or artifact a task needs, such as a training dataset reference, a model artifact, or a numeric threshold. Output: A value or artifact produced by a task for later tasks to consume. Dependency edge: A connection from a producing task to a consuming task. It says the producer's required output must become available before the consumer can use it. In Kubeflow Pipelines (KFP), small values such as integers, strings, and floats are generally passed as parameters . Larger structured items such as datasets and trained models are passed as artifacts with an associated path, URI, type, and metadata. Two component tasks do not share ordinary Python memory. The orchestration platform manages parameter transfer and artifact access between their separate executions. A beginner friendly example: delivery time prediction Imagine a company retraining a food delivery ETA model. It uses two branches: one prepares training features; another calculates data quality statistics. What the edges mean: 1. load orders outputs the same versioned dataset artifact to prepare data and check quality. 2. prepare data creates a processed dataset artifact. train model cannot consume it until it exists. 3. train model outputs a model artifact for evaluate. 4. check quality outputs a small Boolean parameter. In this simplified design, evaluate receives it alongside the model and must reject a failed check. 5. evaluate produces a metrics artifact for inspection or later approval. Production nuance: If data quality validation is a hard safety gate, it is usually better to make the validation task fail on invalid data or explicitly condition downstream training. Merely passing quality passed=False to the final evaluator would waste training resources, and does not automatically prevent training. How are the links expressed in code? Here is a simplified Kubeflow Pipelines SDK pipeline declaration. Assume the components load orders, prepare data, train model, and evaluate have already been implemented with compatible typed interfaces. The named outputs shown are illustrative contracts . Real components must declare them with matching artifact or parameter types. This is a pipeline composition example, not a standalone program, because those component definitions are not included. Notice that you never manually tell train model to wait for prepare data. Passing prepared.outputs["features"] as an input creates the dependency automatically. Parameters versus artifacts Suppose a preprocessing task reports row count=100000 and creates a processed CSV file. row count: int is a parameter: the orchestrator can pass this small value directly. Processed dataset is an artifact: the file is stored in an artifact location, while the platform tracks how downstream tasks consume it. Similarly, a training task might emit: training seconds: float as a parameter. Trained model file as a Model artifact. Important: Artifact metadata or URI alone does not guarantee downstream code can read the data. Artifact storage, permissions, runtime compatibility, and valid content still matter. What can execute in parallel? If two tasks depend only on load orders, they may run independently after the dataset is ready: A downstream task receiving outputs from both branches must wait for both required dependencies. If a task needs to wait for another task without consuming an output , orchestration systems such as KFP also support explicit control dependencies (for example, .after(...)). The distinction is: Data dependency: “I need the artifact you produced.” Control dependency: “Run only after that task finishes.” The next curriculum questions explore the graph structure and execution order in greater depth. How should artifacts be tracked? Each important artifact should be associated with details such as: The producing task and pipeline run ID. Dataset or model version and storage location. Creation time, relevant schema, and compatibility requirements. Upstream input artifact identifiers. Evaluation metrics or validation status when applicable. KFP uses ML Metadata to track parameter and artifact lineage, letting engineers trace which upstream inputs contributed to an output. This is useful when debugging an unexpected model result. Common mistakes Assuming a plain local path is shared across tasks: Separate container executions may run on different machines. Use a supported artifact store or shared workspace. Passing a huge dataset as a string parameter: Use an artifact reference instead. Using only task names to imply a dependency: A meaningful ordering requirement must be represented by data or control edges. Overlooking branch synchronization: A downstream task must receive or explicitly wait for all the results it needs. Treating successful task completion as model approval: Artifact availability and quality gates are separate concerns. In short: Components become a pipeline dependency graph when their outputs are connected to downstream inputs . Parameters carry small values; artifacts carry ML datasets, models, and reports. Those connections determine which tasks can run now, which must wait, and how results can be traced through the workflow.

What is a directed acyclic graph (DAG) in ML workflow orchestration?

A directed acyclic graph (DAG) is a way to describe an ML workflow as tasks connected by one way dependencies, without a path that loops back to its starting task . An orchestrator uses this structure to decide which tasks are ready, which can run together, and what should wait. What do the three words mean? Directed: Arrows have a direction. Validate → Train means training depends on validation, not vice versa. Acyclic: Following the arrows cannot return to a task already visited. A cycle such as A → B → C → A creates an impossible prerequisite loop. Graph: Tasks are nodes, and connections between them are edges. In orchestration, a DAG describes control flow ordering . It does not necessarily move the full training dataset between tasks; datasets and artifacts may live in external storage. Beginner friendly practical example A food delivery company runs a pipeline to train a delivery time prediction model. Its workflow is: There are six tasks . After validation succeeds, Train candidate and Compute baseline can be eligible to run in parallel because neither depends on the other. Evaluation must wait until both have completed successfully under this example's dependency rules. If validation fails, the dependent training and baseline tasks should not proceed as though the data were valid. The arrows describe which task must finish before another becomes eligible . They do not dictate where a model is stored or guarantee that hardware is immediately available. Why must the graph be acyclic? Suppose we accidentally define: Then training waits for evaluation, while evaluation waits for training. Neither can start. A workflow orchestrator generally rejects such cycles or cannot construct a valid execution order. This restriction does not mean a production ML system cannot retrain repeatedly. Each scheduled daily run can execute a new instance of the same DAG . Retraining based on feedback belongs to a later run or an explicitly modeled outer orchestration process, not a cycle within one DAG. How does orchestration use the DAG? An orchestration engine typically: 1. Parses the graph to identify tasks and their prerequisite relationships. 2. Checks readiness: A task can become eligible after its upstream requirements are met. 3. Schedules eligible tasks on available workers or infrastructure. 4. Records states: Running, succeeded, failed, skipped, or waiting. 5. Applies execution rules: Retries, timeouts, triggers, and workflow level schedules when configured. In Apache Airflow, a DAG provides workflow structure and scheduling configuration; individual tasks perform the actual work. In ML pipeline systems such as Kubeflow Pipelines, the DAG can describe containerized tasks connected through input/output relationships. Parallel ready does not mean simultaneously running. A cluster with one free worker may execute eligible branches one after another. A small Python shaped example The following pseudocode illustrates dependency edges rather than a complete runnable Airflow or Kubeflow program: An Airflow implementation could express dependencies with , but creating a real workflow also requires defining the task objects and DAG configuration. Different frameworks have different APIs and dependency syntax. DAG vs. a simple checklist A checklist describes steps in human language. A DAG makes execution constraints explicit and machine readable : It shows branches that can run independently. It lets orchestrators block downstream tasks when prerequisites fail. It makes the critical path through the workflow easier to understand. It provides a structure for execution tracking and operational alerts. A DAG can also be entirely linear. Branching is useful but not required . What is different from the previous concept? Question 53 explained how inputs, outputs, and artifacts create dependencies between pipeline components . This lesson concentrates on the graph's formal properties and orchestration consequences : directed arrows, absence of cycles, readiness, safe parallel branches, and repeated DAG runs. In short: A DAG is a one way, cycle free blueprint for executing an ML workflow. It makes task prerequisites explicit so an orchestrator can schedule work in a valid order, exploit available parallelism, and handle failures consistently.

How do task dependencies determine pipeline execution order?

Task dependencies determine which ML pipeline steps may start and which must wait. An orchestrator checks each step's upstream requirements and only schedules it when its configured conditions are met. The previous lessons introduced pipeline artifacts (Question 53) and the DAG structure (Question 54). Here the focus is runtime scheduling : when tasks become eligible, what can run concurrently, and what happens if a prerequisite fails. A beginner friendly example with numbers A food delivery company trains a model for predicting delivery times. Assume five tasks and their estimated durations: 1. Extract data (A): 2 minutes. 2. Validate data (B): 3 minutes; depends on A. 3. Train model (C): 8 minutes; depends on B. 4. Calculate baseline metrics (D): 4 minutes; depends on B. 5. Evaluate candidate against baseline (E): 2 minutes; depends on both C and D . The dependency relations are: If workers are available and the duration estimates hold: Minute 0–2: A runs. All others wait. Minute 2–5: B runs after A succeeds. Minute 5–13: C runs after B succeeds. Minute 5–9: D also runs after B succeeds; it can run in parallel with C. Minute 13–15: E runs because both C and D are complete. Total time: 15 minutes with enough worker capacity. If the tasks ran strictly one after another, the total would be 19 minutes (2+3+8+4+2). The four minute difference comes from allowing the independent C and D branches to overlap. The critical path is A → B → C → E (15 minutes); speeding up D alone would not make the entire pipeline finish sooner while C remains the longer prerequisite. What makes a task ready to run? An orchestrator generally considers: Upstream completion: Have the required predecessor tasks finished in the appropriate states? Trigger rule: Does the task require all predecessors to succeed, or a different condition? Capacity: Is a worker slot or appropriate compute resource available? Other scheduling controls: Do pool limits, concurrency quotas, scheduling windows, or task state restrictions permit the run? A task can therefore be eligible according to dependencies but still remain queued while waiting for a worker. Dependencies determine valid order; resource availability determines actual start times. How are dependencies written? In Apache Airflow , the operator can declare upstream to downstream relationships: These lines are a dependency fragment , not a complete runnable DAG: the task objects and DAG must be defined separately. In Kubeflow Pipelines , passing a task's output into another component's input can establish ordering automatically. For a control dependency without data passing, a task can explicitly depend on another task (for example, with an .after(...) relationship in the relevant DSL). A useful rule is: Add an edge only when one task really must precede another. A needless dependency forces otherwise independent work to run in sequence. What happens when a prerequisite fails? Suppose validation B fails at minute 4. Training C and baseline D should not start with invalid data. Under a typical all upstream success rule, downstream evaluation E also cannot run successfully. Some pipelines intentionally configure exceptions: An alerting task may run after failure. A cleanup task may run regardless of whether earlier steps succeeded. A reporting task may run when one of several optional branches completed. In Airflow these behaviors are configured using trigger rules , such as all success or all done. all done means upstream tasks finished in terminal states; it does not mean all succeeded. A workflow should choose rules deliberately so that a model is never approved merely because its evaluation task became executable after a failure. Retries and exponential backoff are separate reliability controls covered in the next concept. How do you design good execution dependencies? 1. Identify inputs and outputs for each task, then connect genuine prerequisites. 2. Separate independent work so it can run concurrently when resources permit. 3. Set explicit success conditions for quality critical steps, especially model evaluation and promotion. 4. Test failure cases to confirm that unsafe downstream steps are blocked. 5. Monitor queue time separately from processing time; slow pipelines may be waiting for resources, not running slowly. In short: Task dependencies create a partial order, not necessarily one fixed sequential list. They prevent tasks from starting before their prerequisites, expose opportunities for parallelism, and combine with trigger rules and available workers to determine actual execution.

How do retries and exponential backoff work in ML pipelines?

Retries let an ML pipeline run a failed task again when the failure may be temporary. Exponential backoff increases the wait between attempts so a busy or unavailable service has time to recover. This is useful for intermittent storage timeouts, temporarily unavailable workers, and throttled API calls. It is not a way to fix invalid data, missing permissions, or broken code. A beginner friendly example A food delivery company runs a nightly pipeline that downloads new orders from object storage before training a delivery time model. At 2:00 AM, the download task encounters a temporary storage service timeout. Immediately repeating the request many times could worsen the outage, so the team configures 3 retries after the first attempt, with a 10 second initial wait and a backoff multiplier of 2 . Using the illustrative formula delay before retry n = min(10 × 2^(n−1), 60) seconds the scheduled attempts are: Initial attempt: Starts at 0 seconds; suppose it fails immediately. Retry 1: Wait 10 seconds; starts at 10 seconds. Retry 2: Wait 20 more seconds; starts at 30 seconds. Retry 3: Wait 40 more seconds; starts at 70 seconds. Thus 3 retries means up to 4 total attempts , not 3 total attempts. If each attempt also takes time before failing, actual start times are later. The delays total 70 seconds in this simplified scenario, excluding execution and scheduler waiting time. A retry policy can also set a maximum delay ; with this example's 60 second cap, any later exponential wait would be limited to 60 seconds. Why not retry instantly or forever? Immediate repeated failures create a retry storm : many tasks can overwhelm the same object store, feature service, or model registry as it recovers. A bounded policy reduces that pressure: 1. Initial retry delay: For example, 10 seconds. 2. Multiplier: Usually 2 for conventional exponential backoff. 3. Maximum retries: Stop after a fixed number of additional attempts. 4. Maximum delay: Prevents waits from growing without bound. 5. Jitter: Add a small random variation so hundreds of workers do not retry in lockstep. The formula above is an illustration , not a promise of exact timing in every orchestrator. Scheduler load, jitter, and framework specific backoff algorithms can alter retry times. Which ML errors should be retried? Usually reasonable to retry, within limits: A network connection reset while fetching an immutable dataset. HTTP 429 or throttling from a model registry, subject to a provided Retry After response. A short lived service outage or an interrupted worker. Usually fail fast: Missing required dataset columns or an incompatible schema. A malformed model configuration. Invalid access credentials or permission denial. A deterministic code error reproduced on every attempt. A task should report the actual failure category instead of blindly retrying every exception. Rate limit responses and service documentation may prescribe waits different from a generic exponential formula. What does Airflow configuration look like? The following Airflow task decorator uses the standard retry controls: This is a minimal illustrative DAG ; an actual ML task must implement the download logic and distinguish retryable from permanent errors. Airflow supports retry policies for more selective per exception handling in current releases. Important: Airflow's exact exponential backoff timing can differ from the simple 10/20/40 second teaching example. Do not assume its scheduler implements that formula verbatim. Why does retry safety matter in ML? Consider a training task that writes model latest.pkl and registers a model. It successfully registers the model but crashes before reporting completion. If the orchestrator retries the whole task, it may create two registry entries or overwrite an artifact unexpectedly. A retry is safe only when repeating the work has a controlled outcome. Helpful approaches include: Use stable run identifiers: Associate outputs with the workflow run and task attempt or a deterministic artifact identity. Prefer immutable inputs and versioned outputs: Avoid silently replacing datasets and model versions. Make registration idempotent: Repeatedly registering the same logical result should not create duplicates. Commit outputs deliberately: Write temporary artifacts first, validate them, and publish/promote only after success. Test partial failure: Simulate a crash after the external write but before task completion, then verify retry behavior. The next question examines how to decide whether a pipeline step is retry safe in more detail. What happens after all retries fail? When attempts are exhausted, the task becomes failed and the orchestrator applies its configured downstream rules. Quality critical actions—such as promoting a newly trained model—should not proceed as though a required input task succeeded. Monitor attempt count, delay, final failure cause, and total task latency . A pipeline that eventually succeeds only after frequent retries may still be unhealthy. Question 55 explained how dependencies govern the order of tasks. Retries govern repeated attempts of one task after a failure , while its dependency position generally remains unchanged. In short: Use bounded retries with exponential backoff and jitter for temporary failures; fail fast for permanent errors; and design ML task side effects so retrying cannot corrupt artifacts or duplicate releases.

How do you decide whether an ML pipeline step is retry-safe?

An ML pipeline step is retry safe when running it again after an interrupted or failed attempt does not introduce duplicate effects, corrupt results, or accidentally publish an incomplete model. The key question is not just “Can the code run again?” It is: “What if the first attempt already changed the outside world before the orchestrator learned that it failed?” This lesson builds on idempotency (Question 29) and retry timing (Question 56), but focuses on a step by step decision procedure for individual ML pipeline operations and partial failures . 1. Identify everything the step changes Before enabling automatic retries, trace the step's observable effects: Reads: Does it read a fixed dataset snapshot or the newest live data? Writes: Does it create or overwrite files, database rows, experiment records, checkpoints, or model artifacts? External calls: Does it register a model, launch an expensive training job, send a notification, or promote a model? Downstream visibility: Can another task read partially written output before the step succeeds? A read only task using immutable input is often relatively safe to rerun. A step that launches jobs, appends records, or changes deployment state needs more protection. 2. Beginner friendly partial failure example A food delivery company has a pipeline step that saves the latest delivery time model and registers it. Suppose its intended operation is: 1. Upload a trained model artifact. 2. Register that artifact as model version v12 . 3. Tell the orchestrator the step succeeded. The upload and registration actually succeed, but the worker crashes before reporting success . When the orchestrator starts a second attempt: Unsafe implementation: Uploads another copy and creates an extra registry version, v13 , even though the underlying model has not changed. Retry safe implementation: Uses a stable logical release key, for example delivery time / training run 842 / candidate. It finds the existing artifact and registry entry, verifies they match the expected content, and returns the same result . A retried attempt has a new attempt number but should keep the same logical operation identity . A genuinely new training run gets a different identity. 3. Apply a five question retry safety test For every ML pipeline step, ask: Question A — Are inputs stable? Can both attempts use the same dataset snapshot, model version, configuration, and relevant code? If the step silently reads “latest,” the second attempt could produce a different result. Question B — What if the step fails at each write? Pretend the process stops just before or just after each database update, object upload, or API call. Can you detect what already completed? Question C — Are repeated writes deduplicated? Use a unique key, upsert, conditional write, or external API idempotency token where supported. An ordinary blind INSERT or “create new version” request may duplicate side effects. Question D — Can a partial output become visible? Write to an isolated staging destination first. Validate the completed artifact and then publish it via a documented commit or promotion step. Object storage and registry operations do not automatically form one atomic transaction . Question E — Is completion verifiable? Can a retry query the authoritative store to distinguish “never started,” “partly finished,” and “committed”? A successful HTTP timeout is ambiguous: the server may have completed the request even though the client did not receive confirmation. If any answer is uncertain, do not automatically assume that the step is safe to repeat . 4. A small decision table in words Validate an immutable dataset: Generally retry or rerun , provided the validator has no non idempotent side effects. Upsert features for a fixed partition: Often retry safe if a unique key, transaction, and consistent input snapshot are enforced. Upload a model to a unique content addressed path: Often retry safe if the same bytes are expected and verified. Register a model version through a create only API: Unsafe by default unless duplicate requests can be reconciled or the API supports an idempotency key. Charge an external service, send an approval notification, or promote production state: Treat as high risk ; use a deduplication/outbox or transactional workflow with reconciliation and suitable human approval. These are conditional judgments, not universal guarantees. 5. Pseudocode: reconcile before writing This is an illustrative pattern; registry and artifact store stand for services with appropriate API support: This requires real atomic uniqueness support at create once. A find by release key check alone is insufficient: two concurrent attempts might both see “not found” and race to create duplicates. Also verify the actual artifact checksum rather than trusting an unvalidated identifier. 6. Test failure at the worst possible point A useful fault injection test: 1. Run the step until after the artifact upload , then kill the worker. 2. Restart the same logical step. 3. Check that one correct artifact and one logical registry entry exist. 4. Repeat the test with a failure after registry creation but before acknowledging success . 5. Start two duplicate attempts concurrently and verify uniqueness controls. Measure both correctness and unexpected cost . A training task that takes hours may be logically safe to rerun but economically inefficient; checkpoints or recovery of an existing job may be preferable. 7. Decide: retry, resume, reconcile, or stop Retry from the beginning: For short operations with stable inputs and safe outputs. Resume: For long training tasks with validated checkpoints and reproducible continuation. Reconcile: For ambiguous external effects; inspect the external system before creating anything new. Stop and investigate: When the operation might have changed production state or cannot prove what happened. Apache Airflow's best practices guidance recommends treating tasks like database transactions, avoiding incomplete outputs, and preferring upserts over duplicate creating inserts. In short: A pipeline step is retry safe only when repeated attempts have a controlled, verifiable outcome , even after a crash immediately following a side effect. Design stable identities, atomic uniqueness, staged publication, and failure tests before switching automatic retries on.

What is pipeline caching and how are cache keys determined?

Pipeline caching lets an ML workflow reuse the output of an earlier successful task instead of executing the same computation again. A cache key is the identity used to decide whether a previous result is suitable for reuse. Caching saves time and compute, especially for expensive steps such as feature preparation. But a cache hit is correct only when all inputs that affect the result are accounted for . Beginner friendly example A food delivery company trains a delivery time prediction model each day. Its preprocessing step takes 12 minutes to clean orders and compute features. The step uses: Dataset snapshot: orders 2026 10 20 Preprocessing code version: feature v3 Configuration: distance unit=km The pipeline runs twice with exactly these inputs and the same compatible component definition. First run: No matching result exists. Preprocessing executes for 12 minutes, stores features v3, and records metadata about its inputs and output. Second run: The platform finds a suitable prior execution and reuses the saved feature artifact. Assuming artifact access is fast, it avoids nearly 12 minutes of repeated computation . Third run: The dataset changes to orders 2026 10 21. A correctly designed cache identity changes, so preprocessing executes again rather than reusing features for the wrong day. If the feature code or a behavior changing parameter changes, the previously saved output should also become ineligible. What goes into a cache key? A useful conceptual fingerprint is: This is a teaching model , not the exact algorithm used by every platform. Tools may include different fields and may support custom keys. For a preprocessing step, important fields might be the input dataset version, transformation code or component image, feature definitions, and values such as missing value strategy=median. A cache key is not merely a convenient filename. It represents an assertion that the previous output is equivalent to what this execution would produce . Cache hit versus cache miss Cache hit: An eligible earlier execution with a matching identity is found, its outputs are still available and accessible, and the platform reuses them. Cache miss: No suitable result exists, so the task runs and may populate the cache. Cache invalidation: Relevant inputs, code, parameters, or an explicit policy change cause the platform to stop reusing an earlier result. For the food delivery example, changing distance unit=km to distance unit=m must not accidentally reuse the previous feature artifact. Otherwise the model might train on values in the wrong units. A small Kubeflow Pipelines example In Kubeflow Pipelines, caching can be configured per task: When the component definition and inputs match a cached execution, Kubeflow Pipelines can reuse its previous outputs if still available. The platform's own execution and cache metadata determine eligibility; your code does not need to manually calculate a hash. Some Kubeflow versions support custom cache keys . Be careful: custom keys may override automatic fingerprinting of inputs and image details. A poorly chosen constant custom key can incorrectly merge executions that should remain separate. When is caching unsafe or unhelpful? 1. Hidden changing inputs. The task takes snapshot id="latest" but reads new data on every run. Its declared parameter stays the same while actual data changes. Use an immutable dataset identifier or version. 2. Untracked external state. The task calls a changing API or database but the cache identity does not include a relevant snapshot or version. The cached result may be stale. 3. Random or nondeterministic work. If a training step must produce a fresh independent experiment, reuse may violate the objective. Where reproducibility matters, capture seeds and other behavior changing conditions; consider disabling caching for intentionally fresh runs. 4. Side effects. A step sends a notification or performs an approval action that must happen once per workflow run. Reusing its stored outputs may skip that effect entirely. 5. Missing artifacts. A matching cache record is not useful if the output files were deleted, expired, or are inaccessible. What should an MLOps team do? 1. Cache expensive, repeatable transformations with stable inputs and well defined outputs. 2. Version input datasets and component definitions ; do not rely on mutable latest pointers. 3. Expose behavior changing values as explicit parameters , not hidden global settings. 4. Verify cache hit correctness by changing one relevant input and confirming the task reruns. 5. Track hit rate, saved runtime, and invalidation events , and test stale output scenarios. Caching and retries solve different problems. Retries rerun an attempted step after a failure; caching may skip execution by reusing a suitable prior successful result. Question 57 focused on preventing harmful duplicate side effects from retries. This lesson focuses on safely deciding whether existing computational work can be reused. In short: Pipeline caching saves time by reusing previous task outputs. Correct cache keys must distinguish every meaningful version of a task's inputs, logic, and parameters; otherwise the optimization can silently produce stale or incorrect ML artifacts.

When should an ML pipeline skip an unchanged step?

An ML pipeline should skip an unchanged step when its previous output is still correct, available, and fresh enough for the current run . Skipping saves unnecessary computation, but it must be an explicit correctness decision—not simply “the code has not changed.” Question 58 explained how cache keys identify reusable computations . This lesson answers the next operational question: When is reuse appropriate, and when must a task run anyway? A beginner friendly example A food delivery company trains a model to predict delivery times. Its workflow has four steps: 1. Extract an approved orders snapshot. 2. Prepare features ( 12 minutes ). 3. Train the model ( 25 minutes ). 4. Evaluate and approve a candidate ( 5 minutes ). On Monday, the feature step processes dataset orders v10 with preprocessing version features v3 and produces a validated artifact. On Tuesday, the team wants to retrain using the same orders v10, unchanged features v3, and the same feature settings. Decision: The feature step can reuse Monday's output, provided the artifact still exists, is accessible, and has not expired under the pipeline's freshness policy. This saves 12 minutes of feature computation. Training can still run if the team deliberately wants a new experiment. On Wednesday, new orders arrive as orders v11. Even with unchanged preprocessing code, the input data differs. Do not skip feature preparation: the new dataset needs new features. A five part skip decision A step is a good candidate to skip only when all these conditions hold: 1. Its meaningful inputs are unchanged. Compare stable dataset or artifact versions, parameters, and relevant code or component identity. A mutable pointer such as latest is not evidence that the underlying data is identical. 2. Its previously produced output is usable. Confirm the saved artifact or materialized table still exists, is readable, and passed the expected validations. A metadata record pointing to a deleted file does not justify skipping. 3. The output is fresh enough. Some results are reusable indefinitely for an immutable dataset; other results, such as hourly feature statistics, must meet a time based freshness requirement. 4. No required action would be lost. A step that sends a daily compliance report, updates an external deployment, or collects a new observation has a side effect that may be required every run. Skipping its computation or reusing an old output may silently omit that action. 5. The reuse is allowed by the workflow's policy. Some steps are deliberately marked always run—for example, a newly required data quality check, evaluation on today's holdout set, or scheduled drift inspection—even if upstream computational outputs have not changed. If any condition fails or cannot be verified, rerun or reconcile the step instead of automatically skipping. Three different reasons a step might not execute Do not treat these situations as interchangeable: Reused from cache: A previously successful execution's eligible outputs replace a new computation. Downstream tasks can often consume those outputs. Conditionally skipped: A branch or policy says the task should not run at all, such as an optional training branch when no new approved data arrived. Downstream tasks must be designed for that absence. Blocked or failed: A required prerequisite failed or the output is missing. This is not a successful optimization; downstream promotion should not proceed as if valid output existed. Different orchestrators represent these states differently. Always check whether the downstream task actually received valid artifacts. A practical policy example Suppose a training pipeline runs daily. The team defines these rules: Feature preparation: Reuse if dataset snapshot, feature code, and parameters are unchanged and the artifact is available. Model training: Reuse only for a replay of the same logical experiment ; run again if trying a new seed, hyperparameter set, or approved dataset. Data freshness check: Re evaluate against today's freshness deadline; an old successful check cannot prove that today's data arrived. Model promotion: Never automatically treat a previously completed promotion as approval of a new model. Check current evaluation and release policy. A compact implementation neutral decision function is: This is decision logic , not a complete scheduler or a substitute for platform specific cache metadata. How do tools support this? Kubeflow Pipelines can reuse successful component outputs for matching inputs when caching is enabled and outputs remain available. For a task that must execute on every run, disable caching: This assumes evaluation task is an already created Kubeflow pipeline task. Disabling caching means it executes when reached; conditional branches and upstream failure rules can still prevent it from running. Dagster can reason about asset code and data versions so assets are rematerialized only when necessary. Data freshness checks help identify outputs that are too old for their consumers even when no obvious input change event is visible. A common mistake: “Nothing changed, so everything can be skipped” A pipeline's source code commit may be unchanged while its input snapshot, external reference data, model evaluation window, business rule, or freshness requirement has changed. Likewise, a task called fetch current prices may have identical parameters each day but return different prices. If the pipeline reuses yesterday's result solely because the task code is unchanged, the output is stale. In short: Skip unchanged ML steps when you can prove that their outputs are still equivalent, available, fresh, and safe to reuse. Cache keys help identify matching work (Question 58); skip policies decide whether reuse is operationally correct .

How do you parameterize training pipelines across datasets and environments?

Parameterizing an ML training pipeline means defining the workflow once while supplying run specific values —such as dataset snapshot, environment, training hyperparameters, and artifact destination—when the workflow starts. Instead of maintaining separate scripts for development, staging, and production, the same reviewed pipeline code runs with different validated inputs. The goal is reuse with traceability , not hiding differences between environments. Beginner friendly example: one pipeline, three runs A food delivery company trains a delivery time prediction model from order records. It uses one pipeline with these parameters: dataset uri: an immutable training dataset snapshot. environment: dev, staging, or prod. learning rate: a training hyperparameter. epochs: number of training passes. output uri: approved destination for the resulting model artifact. Illustrative run configurations: The pipeline definition stays unchanged . Development trains quickly on a small approved sample, while production trains on an approved snapshot with a separately authorized artifact destination. These storage URIs are examples; a real environment needs credentials or workload identity with appropriate access. A configuration file does not grant access by itself. What belongs in parameters? Use parameters for values that genuinely vary from run to run: 1. Data identity: Dataset URI, partition, snapshot ID, label version, or evaluation window. 2. Training behavior: Hyperparameters, random seed, model family, and training mode. 3. Environment selection: A constrained choice such as dev, staging, or prod. 4. Artifact identity: Model output prefix, experiment name, or registry namespace. 5. Execution options: Whether to run optional evaluation or use a designated resource profile, if those choices are permitted. Keep shared feature logic and task definitions in versioned code. Do not turn every implementation detail into a user supplied switch: too many knobs create untested combinations. Practical Kubeflow Pipelines example Kubeflow Pipelines allows typed pipeline parameters to be passed into components. This example illustrates typed run inputs ; the component is a placeholder, not a trained model. In a real pipeline, environment should be validated and used to select an allowlisted resource profile or artifact location, rather than accepted and then ignored. The application should also verify permissions before writing. A run submission can pass values from a reviewed configuration file. Keep the compiled pipeline version fixed while comparing runs; supply the run inputs at execution time. Validate parameters before expensive work begins A production pipeline should enforce rules such as: environment must be one of dev, staging, and prod. epochs must be a positive integer within the team's permitted range. learning rate must be finite and within an approved numeric interval. dataset uri must reference an approved, immutable snapshot that the job can actually access. output uri must be within the authorized storage namespace for that environment. A staging run must not be allowed to overwrite production model artifacts. Do not trust a plain environment=prod string as authorization. Enforce access with IAM/RBAC and resource policies. Configuration validation and access control solve different problems. Record the effective configuration For every run, record: Pipeline code or compiled definition version and container image digest. Resolved dataset snapshot and model/preprocessing code versions. Hyperparameters and random seed. Environment, approved output location, and the actual run identifier. Evaluation metrics and the resulting model artifact identifier. This lets engineers answer: “Why did the production model differ from the staging model?” The answer might be the input dataset, epochs, random seed, or a code revision—not an invisible machine specific default. Avoid logging passwords or other secrets alongside the configuration. Use secret management and workload identity for sensitive access. When should you create a separate pipeline? Parameterization works best when the steps are conceptually the same . If one use case requires fundamentally different validation, model architecture, data governance, or approval stages, a separate pipeline or clearly defined sub workflow may be safer than dozens of conditional flags. Airflow also supports validated runtime Params, and Kubeflow supports pipeline input parameters; exact syntax differs, but the same design principle applies. Question 59 focused on whether an unchanged step's output can be reused . This lesson concerns how one pipeline definition handles different datasets, environments, and training configurations ; changes to these values can also affect caching, but skip decisions are a separate concern. In short: Parameterize dataset identity, hyperparameters, environment, and artifact locations; validate every run's values; enforce storage permissions; and record the effective settings with the resulting model.