MLOps
MLOps applies software engineering and operations discipline to machine learning. It covers reproducible training, model registries, deployment, monitoring, governance, and incident response.
How do you schedule recurring ML pipeline runs?
Recurring ML pipeline scheduling means configuring an orchestrator to create new pipeline executions at a planned cadence—such as every night, every Monday, or once per month—without manually starting each run. A schedule answers when a run should be created . The pipeline definition answers what work the run performs . Reliable scheduling also requires deciding which data period a run processes , what happens when yesterday's run is still active, and whether missed runs should be replayed. A beginner friendly practical example A food delivery company trains a delivery time prediction model every morning after the previous day's orders have arrived. Its requirement is: Run at 02:00 UTC every day, process the previous complete UTC calendar day, and avoid overlapping training jobs. For instance, the scheduled execution at 02:00 UTC on October 26 should process orders from October 25, 00:00 UTC up to (but not including) October 26, 00:00 UTC . This is a business rule the training code implements; a 02:00 schedule alone does not automatically choose the correct input partition. The five field cron expression is: Important: Cron is interpreted in the scheduler's configured timezone. Using UTC avoids daylight saving changes, but the business may require an explicit local timezone instead. A practical Apache Airflow example This minimal Airflow 3 style example uses a trigger timetable so the intent is easy to see: run daily at 02:00 UTC. The task deliberately derives yesterday's partition from the scheduler supplied logical date rather than from the machine's changing wall clock. This is a teaching example , not a fully implemented training pipeline. The actual training step must validate that the dataset for the selected date is complete and enforce a safe model promotion policy. With a trigger timetable, logical date represents the scheduled trigger point. Other Airflow timetables use data intervals and can assign a different meaning to logical date. For production code, use the selected timetable's documented logical date/data interval semantics rather than assuming every timetable is interchangeable. Four scheduling decisions that matter 1. Cadence: how often? Choose daily, hourly, weekly, or monthly based on how quickly the data or model changes and what operations can support . A high frequency schedule with an hour long training job may create unnecessary pressure. Model updates do not have to be published after every scheduled training run. 2. Timezone and input data interval: what time and which records? Define both the schedule timezone and the data window . A run starting at 02:00 should not accidentally train on incomplete data for the current day. If data is delivered late, use an explicit completeness check or delay until the source data is ready. 3. Overlap: what if yesterday's run is still running? Airflow's max active runs=1 limits concurrent DAG runs. In Kubeflow Pipelines, a recurring run configuration can also limit concurrent runs. This protects expensive training infrastructure and shared outputs, though the orchestrator may queue later work. Decide whether delayed runs should wait, be skipped, or be handled manually. 4. Catchup and missed schedules: what if the scheduler was paused? With catchup=False, Airflow normally avoids automatically creating every historical scheduled run since the start date. With catchup enabled, older eligible intervals can generate runs, potentially leading to many costly training executions. Backfills for earlier dates should be a conscious decision with pinned input snapshots; they are covered in a later lesson. How is recurring scheduling configured in Kubeflow Pipelines? Kubeflow Pipelines supports recurring runs with either a cron trigger or a periodic interval. A recurring run stores a repeatable pipeline/run configuration and creates executions according to the trigger. Teams can enable or disable a recurring run and configure how many copies may run concurrently. This is similar in intent to an Airflow DAG schedule, though the APIs and execution semantics differ. What should be monitored? Expected vs. created runs: Was today's run actually scheduled? Start delay: Did it wait for a worker or resource quota? Completion status and duration: Did the run finish before its operational deadline? Data completeness: Did it process the intended snapshot or partition? Overlaps and skipped/missed dates: Did concurrency controls or a paused schedule leave gaps? Release outcome: Was a candidate only trained, or actually approved and deployed? A successful scheduler tick is not proof that training completed or a model was safe to deploy. Scheduling versus the previous lesson Question 60 covered how one pipeline accepts different dataset IDs, environments, and hyperparameters . Here those inputs can remain fixed or be resolved for each scheduled date; the key new topic is automatic run creation, calendar semantics, time zones, catchup, and concurrency . The next question compares recurring time triggered runs with pipelines started by events , such as an incoming dataset. In short: Schedule ML pipelines with an explicit cadence and timezone, make the processed data window unambiguous, bound concurrency, handle missed intervals deliberately, and monitor each run from scheduling through completion.
How do event-triggered and scheduled ML pipelines differ?
An event triggered ML pipeline starts in response to a meaningful event, such as a validated dataset arriving or a new model candidate being registered. A scheduled ML pipeline starts according to a clock based rule, such as every morning or every Monday. The difference is what initiates a run , not the kind of ML code it executes. Both approaches can run the same validation, feature preparation, training, and evaluation tasks. A beginner friendly example A food delivery company retrains its delivery time model on completed orders. Scheduled approach: Start training every morning at 02:00. If yesterday's orders are still being transferred, the run must wait, fail a completeness check, or use the last approved snapshot according to policy. Event triggered approach: Start training when the data platform publishes an event that says the completed orders snapshot for a specific date is validated and ready . If it is ready at 01:40, the pipeline can start then; if it is not ready until 03:15, the run starts later. A raw “file uploaded” event is not necessarily sufficient : large datasets may be uploaded in several parts. Prefer a trustworthy ready or manifest committed event indicating the entire expected snapshot is available. The difference in practice Scheduled pipelines Trigger: Calendar time or fixed interval. Best for: Periodic retraining, daily reports, recurring quality checks, and work that must happen even when there is no new input event. Trade off: Predictable operational cadence, but a scheduled run may find incomplete or unchanged data. Operational question: “Which data window does the 02:00 run process?” Event triggered pipelines Trigger: Dataset ready notification, successful upstream asset update, message queue event, or controlled API request. Best for: Training whenever validated data arrives, reacting to upstream pipeline completion, or processing newly approved model artifacts. Trade off: Can reduce waiting and avoid unnecessary runs, but requires reliable event delivery, deduplication, and clear readiness semantics. Operational question: “Which exact dataset version caused this execution?” These choices are not mutually exclusive : teams can use events for new data and a periodic schedule for compliance checks or freshness monitoring. A practical event payload An upstream ingestion service can emit an illustrative event after its data quality checks pass: The orchestrator should validate the sender and event schema, check that the referenced snapshot actually exists and is authorized, and start the pipeline with snapshot id as an explicit run input. If the event is delivered twice , both copies should map to the same logical processing identity (for example, orders + snapshot id + pipeline version) so duplicate events do not start unintended duplicate training or model promotions. Events are usually not equivalent to exactly once execution. Durable messaging, retries, and consumer deduplication must be designed together. How does an orchestrator implement the trigger? Apache Airflow supports asset aware scheduling : a producing task can declare that it updates an asset, and a consumer DAG can be scheduled from that asset update. Airflow also supports event driven scheduling based on external signals, using supported asset watchers or API delivered asset events. A minimal Airflow 3 style illustration is: This shows the consumer side only . A trusted producer must emit the corresponding asset update; simply uploading an object to storage does not automatically emit an Airflow asset event. The real consumer also needs to resolve the specific triggering snapshot , potentially from asset event metadata, rather than assume a mutable latest path is safe. AWS EventBridge can route matching events or scheduled rules to Amazon SageMaker AI Pipelines , with configured execution parameters. Kubeflow Pipelines recurring run triggers are based on cron or periodic intervals; external event driven starts require an appropriate integration that submits pipeline runs. A small numerical comparison Suppose validated order snapshots arrive at these times on three days: Day 1: 01:30 Day 2: 02:45 Day 3: 01:50 A 02:00 scheduled training pipeline would have new complete data ready on Days 1 and 3, but on Day 2 it must wait or otherwise handle missing data. A dataset ready triggered pipeline could begin after each snapshot is validated: 01:30, 02:45, and 01:50 respectively (subject to event delivery and worker delays). This does not prove that event triggered processing always finishes faster: queueing, competing jobs, batching, event delivery, and mandatory validation still add latency. Reliability checklist for event triggered runs 1. Trust the trigger: Verify event origin, authorization, and schema. 2. Confirm data completeness: Require a committed manifest, immutable snapshot, or equivalent ready signal. 3. Deduplicate: Use a stable event or snapshot identity; handle replayed messages. 4. Limit concurrency: Avoid launching many costly training jobs for a burst of closely related events. 5. Handle delayed and out of order events: Do not accidentally replace a newer production model with a late older snapshot. 6. Retain a reconciliation mechanism: Periodically compare expected datasets with completed pipeline runs so a lost or misrouted event does not leave a permanent gap. Question 61 focused on how to configure recurring schedules, time zones, catchup, and concurrency . This lesson compares the decision to trigger on time versus an observed change in the world , and the extra correctness requirements that events introduce. In short: Schedule a pipeline when the business needs a predictable cadence; trigger from events when work should respond to a verified change. Use both where useful, and always protect data identity, deduplication, and safe model release.
How do you handle backfills in ML data and training pipelines?
A backfill reprocesses historical ML pipeline periods that were missed, failed, or need recalculation after a bug fix. It is different from a routine schedule: you explicitly select past dates or partitions . Practical example A food delivery company's daily feature preparation job missed October 20, 21, and 22. Data for October 23 onward is already correct. To restore these three days: 1. Identify the three missing logical dates . 2. Use the correct immutable dataset snapshot for each date, never a mutable latest pointer. 3. Recalculate and validate the three partitions. 4. Publish using partition aware upserts or staged atomic updates, preventing duplicate rows. 5. Verify row counts, feature schema, and output versions. 6. Keep historical model retraining separate from automatic production model promotion. If each partition takes 8 minutes , the total processing work is 24 minutes. With two free workers and independent partitions, approximate best case elapsed compute time is 16 minutes : two jobs first, then the last. Startup and resource contention can increase this. Key decisions Backfill window: Specify historical start/end dates and distinguish execution time from the period being processed. Reprocessing policy: Do you only fill missing runs, recreate failed runs, or intentionally recompute previously successful outputs? Code and data versions: Decide whether the backfill reproduces the historical logic or uses a corrected version; document the choice. Compute safety: Limit concurrent runs and protect upstream sources, quotas, and budgets. Data correctness: Check point in time features and label cutoffs to prevent leaking future information into historical training. Release safety: Do not promote a historical model over a newer production model without explicit evaluation and approval. Airflow 3 backfill example For a time scheduled DAG named delivery features daily, Airflow supports bounded backfill runs: Review the output before running the command without dry run. In current Airflow's reprocessing choices, none avoids creating a new run when one already exists for a date, regardless of its state. For previously failed runs, choose the failed policy when a new run should replace the failed execution; completed permits reprocessing completed or failed dates. Airflow does not recreate a currently running or queued run under these policies. Check the installed Airflow version: CLI option names differ among releases. The DAG must have a backfill compatible time schedule. The concurrency limit shown applies to backfill DAG runs; task level worker demand may be greater. How do you verify a backfill? Compare the expected list of dates to completed runs. Confirm every run read the intended historical partition, outputs passed data quality tests, duplicate records were not created, and downstream consumers see complete artifacts. If training was replayed, confirm model versions are isolated from production promotion. Common pitfalls A historical job reads today's data because of an unversioned source path. A corrected preprocessing function accidentally uses labels that became available after the historical cutoff. A partial write followed by a retry duplicates records. Replaying dozens of dates at once overloads infrastructure. An older retrained candidate automatically replaces the live model. Questions 61–62 covered when new runs are scheduled or triggered . Backfills cover safe replay of chosen historical periods . In short: Backfill with explicit dates, fixed inputs, controlled concurrency, idempotent outputs, and validation. Treat historical model promotion as a separate deliberate decision.
How should failed pipeline executions be resumed or replayed?
A failed ML pipeline can be recovered in four ways: retry a task, resume saved progress, selectively rerun affected tasks, or replay the full workflow . The correct choice depends on whether earlier results remain valid and what was already written outside the pipeline. Retry, resume, selective rerun, and replay Retry: Repeat one failed task after an error. It generally starts the task function again unless the application itself loads saved progress. Resume: Continue interrupted work from a valid checkpoint or paused execution state instead of discarding completed progress. Selective rerun: Run failed or affected tasks and their required downstream work, reusing trustworthy upstream artifacts. Replay: Create a fresh execution of the full logical workflow using explicitly identified data, configuration, and pipeline versions. These terms sometimes vary by platform. The operational distinction is what work is repeated and what previous state is retained . Beginner friendly numerical example A food delivery company trains a delivery time model with these steps: 1. Validate input orders: 3 minutes. 2. Prepare features: 15 minutes. 3. Train candidate: 40 minutes. 4. Evaluate candidate: 5 minutes. 5. Register candidate after passing evaluation. Validation and feature preparation succeed. The trainer saves a checkpoint at training minute 30 but its worker crashes after 35 minutes. Three recovery choices have different costs: Retry the training task from scratch: Repeat up to 40 minutes of training while reusing the valid features. The previous 15 minute feature computation need not run again. Resume from the checkpoint: Restore valid model and optimizer state and finish roughly 10 minutes of training, plus loading time. Recovering only model weights may not reproduce the optimizer state or data sampler position. Replay the whole pipeline: Repeat validation, feature preparation, and training. This is appropriate when earlier outputs are unreliable, but is more expensive. These times are illustrative; actual recovery depends on checkpoint compatibility, compute capacity, data order, and restoration overhead. A reliable recovery decision process 1. Diagnose what failed. Find the failed task and attempt, worker logs, exit code, and any external service calls. A timeout does not prove the remote operation did not complete. 2. Check completed artifacts. Confirm the dataset snapshot, preprocessing version, checksums, training configuration, and artifact availability. Reusing an incorrect feature artifact only reproduces a broken result. 3. Locate the earliest untrusted boundary. If training alone failed because a worker disappeared, reuse the validated features. If preprocessing had a bug, regenerate features and downstream results. If input versions are unknown, perform a controlled full replay. 4. Choose the recovery method. Retry short idempotent tasks; resume long running training when checkpoints are complete and compatible; selectively rerun affected tasks when upstream work is valid; replay from the beginning when correctness requires it. 5. Protect side effects. A model artifact upload or registry creation may have succeeded just before a crash. Reconcile that state before repeating the operation. Use stable logical run identities, uniqueness checks, and explicit promotion approval. 6. Verify recovery. Inspect task states, output lineage, final model metrics, and production deployment status. Tool example: Apache Airflow In Airflow, an operator can clear a failed task instance , allowing the scheduler to execute that selected task again. It is possible to include downstream tasks when necessary. Clearing an Airflow task does not reverse actions already performed by its previous attempt, such as inserting database records, uploading model artifacts, or registering candidates. Carefully review which tasks are selected before clearing a group. Airflow task retries, covered in Question 56, are the automated form of repeating a task attempt; recovering a failed run may instead require operator investigation and selective rerunning. Tool example: Amazon SageMaker AI Pipelines SageMaker AI Pipelines supports selective execution of selected steps with compatible artifacts from a reference execution. This can avoid repeating successful and expensive upstream computation. The training code itself must implement saving and restoring usable checkpoints. Merely relaunching a pipeline step does not guarantee that it resumes from the exact batch where it failed. When not to resume The saved checkpoint is incomplete, corrupt, or incompatible with the model or optimizer. The input snapshot changed or cannot be identified. An upstream step produced a wrong artifact. A remote operation may already have changed production state and needs reconciliation. The purpose is a fresh reproducibility test that requires a new full run. Difference from Question 63: A backfill handles chosen historical periods, such as three missing dates. This question concerns recovering one failed execution and deciding which completed steps can be trusted , regardless of its date. In short: Diagnose first, preserve verified upstream work, resume from valid checkpoints when possible, rerun the smallest affected section, and replay the entire pipeline only when necessary.
How do you build pipeline-level observability and run histories?
Pipeline level observability means being able to answer: What ran, when, with which inputs, what happened in every step, and which artifacts were produced? A run history preserves that evidence across executions, not just the latest status. This is different from Question 64: there we chose how to recover a failed workflow; here we build the records and signals needed to understand and compare runs , whether successful or failed. Practical example: delivery time training A food delivery company trains a model daily. Its orchestrator assigns a stable identifier to each workflow execution: The run contains these step records : validate orders: SUCCESS — 30 seconds — 120,000 rows accepted. build features: SUCCESS — 3 minutes — artifact features/orders 2026 10 25 v2. train model: SUCCESS — 12 minutes — artifact models/candidate 842. evaluate model: FAILED — 1 minute — error: evaluation reference dataset unavailable. register model: NOT STARTED — no model registered. Someone opening this run should see the failed evaluation task, its timestamped logs, its missing input, and the affected downstream step. They should not have to search several worker machines to discover which training data was used. What should you capture? 1. Pipeline run identity and context Record pipeline id, unique run id, start/end timestamps, trigger type, effective parameters, dataset snapshot IDs, environment, pipeline definition version, and image digest. Use one run id in logs, metrics, tracing, and artifact metadata. 2. Step execution history For each task capture task id, attempt number, state (queued, running, success, failed, skipped), start/end times, duration, worker identity where appropriate, error category, and link to relevant logs. Distinguish a task waiting in queue from one actively consuming compute. 3. Structured logs Logs should answer what each component tried to do and why it failed. Example: Include correlation IDs and meaningful errors, but do not log credentials, customer records, or sensitive training examples . 4. Operational and ML metrics Operational metrics include total runtime, per step duration, queue delay, task failures, retry counts, CPU/GPU utilization, and successful run percentage. Model metrics can include MAE, RMSE, F1, or another evaluation measure appropriate to the task, alongside the evaluation dataset version. 5. Artifact lineage Connect input dataset → feature artifact → trained model → evaluation artifact. Each artifact should have a durable identity, version or digest, storage location, producing run id and task id, and relevant parent artifact IDs. This makes it possible to determine which data and code produced a particular model . How does run history help? A numerical comparison Three recent runs of the same delivery time pipeline have durations: Run A: 16 minutes, SUCCESS Run B: 18 minutes, SUCCESS Run C: 31 minutes, FAILED during evaluation The historical average elapsed time is (16 + 18 + 31) / 3 = 21.7 minutes (including the failed run). The success rate is 2/3 = 66.7% across this tiny sample. These figures flag something to inspect, but three runs are too few to establish a reliable baseline. Per task durations, queue delays, and logs can show whether Run C spent extra time waiting for GPUs, training, or failing an external lookup. How do tools support this? Apache Airflow: The DAG Runs and Grid views show historical runs, task states, attempts, timing, and task logs. They are useful for operational diagnosis and comparing multiple pipeline executions. Add external logging/metrics retention when task workers are ephemeral. Kubeflow Pipelines: Tracks pipeline runs, component executions, and ML artifacts. Its ML Metadata store records execution and artifact properties and can visualize lineage across runs , including an artifact consumed by more than one run. ML experiment trackers: Systems such as MLflow can complement an orchestrator by recording hyperparameters, evaluation metrics, and model artifacts. Use a shared run correlation ID to join experiment records with pipeline run history. A practical observability checklist 1. Emit the same run ID from every task and store it in all artifacts. 2. Record exact input snapshots, resolved parameters, component code/image versions, and output checksums. 3. Capture task states and attempt counts, including skipped and failed steps. 4. Centralize searchable structured logs, retaining useful error details without secrets. 5. Monitor queue times, runtimes, failures, and data quality or model evaluation metrics. 6. Maintain dataset to model artifact lineage and retain run history according to governance needs. 7. Alert when expected runs are missing, critical tasks fail, outputs are late, or model quality gates are violated. 8. Test the workflow by locating one model artifact and tracing it back to its producing run and source snapshot. In short: Pipeline observability combines run IDs, step states, logs, metrics, and artifact lineage so teams can diagnose failures, explain model provenance, measure reliability, and compare executions over time.
How do you detect and prevent concurrent pipeline run conflicts?
Concurrent pipeline run conflicts happen when two or more workflow executions overlap and compete for a shared resource, output path, dataset partition, model registry entry, or deployment target. Running at the same time is not inherently wrong: the danger is unsafe shared state . Question 65 introduced monitoring run IDs, logs and artifact lineage. Here the emphasis is detecting collisions and designing workflows so overlapping runs do not corrupt results or publish the wrong model . Beginner friendly practical example A food delivery business retrains a delivery time predictor daily. The October 25 training run starts at 01:00 and needs 90 minutes , but a manually triggered run for October 26 starts at 01:30 . Both use an unsafe shared output path: Run A: 01:00–02:30; dataset orders v25. Run B: 01:30–03:00; dataset orders v26. The two runs overlap for 60 minutes . If A uploads at 02:30 and B uploads at 03:00, the final object might be B's. But if uploads or completion order change, an older model can overwrite the newer model. A separate registry task might even read the other run's artifact , producing misleading lineage. A safer layout gives each run its own immutable candidate: Each evaluation consumes its own run scoped artifact . A distinct promotion step decides which validated candidate becomes production, using a protected registry or deployment update rather than overwriting a shared path. Where can runs conflict? Output paths: Two jobs write latest.csv, the same object key, or the same unversioned model file. Data partitions: Two feature jobs concurrently overwrite or append to the same logical date partition. Registry/deployment: Both jobs try to assign the same production alias or update the serving endpoint. Limited resources: Overlapping GPU heavy runs starve one another or exceed quotas. External systems: Duplicate pipeline launches race to create the same remote training job. Mutable state: One job changes shared configuration while another is running. Some overlaps are merely costly; others are correctness or safety bugs . Treat deployment and data corruption as higher priority than reducing run duration. How do you detect conflicts? 1. Attach a unique run ID to every task, log and artifact. 2. Track the logical resource identity , such as dataset=orders, partition=2026 10 25 or model=delivery time, target=production. 3. Detect two active runs claiming the same exclusive resource . Distinct dataset partitions may safely run in parallel. 4. Alert on duplicate writes, optimistic lock failures, overlapping promotions, unusual queue delays, and missing ownership information. 5. Audit the final model registry or storage state to check which run actually produced what is live. A sample structured event: This event is illustrative; orchestration tools do not automatically emit it unless the team adds detection. How do you prevent collisions? 1. Give outputs run scoped identities. Store temporary and candidate outputs under immutable dataset and run IDs. Use stable checksums and record lineage. 2. Constrain run level parallelism. For workflows that must never overlap, set a maximum of one active run. For example, in Apache Airflow: This is a scheduling illustration rather than full model training code. max active runs=1 only restricts runs of that DAG ; it does not prevent an independent DAG, external job, or another service from changing the same resource. 3. Protect shared resources across workflows. Use a transactional database constraint, atomic compare and swap or version check, lease/lock with expiry and fencing tokens, or an equivalent service provided conflict prevention mechanism. Locking must handle worker crashes; an expired lease alone can be unsafe when the old worker is still acting. 4. Separate candidate creation from promotion. Concurrent training may be desirable. Serialize or conditionally authorize production promotion based on evaluations and a controlled release policy. 5. Handle duplicate logical work. Choose a unique identity such as pipeline version + snapshot id + experiment key. Reject or deduplicate simultaneous submissions of the same logical job unless repeated experiments are intended. 6. Limit expensive shared capacity. Airflow pools and task concurrency limits, or Kubeflow Pipelines recurring run maximum concurrency, can restrict load. A GPU capacity limit is not the same as a data consistency guarantee . What if two runs genuinely can run together? If Run A works on the October 25 partition and Run B on October 26, and both use isolated artifacts, they may run concurrently. Keeping all ML pipelines sequential wastes compute. Apply resource specific rules: parallelize independent preprocessing and training, but protect shared dataset partitions and production model aliases. Testing the design Start two runs simultaneously against the same logical dataset and production target in a staging environment. Confirm that candidates stay isolated, at most one authorized promotion wins, the losing run gets a clear conflict outcome, and all objects have accurate run lineage. Then simulate a crash after acquiring a lease or just before publishing. In short: Detect concurrent conflicts by tracking which runs operate on which resources . Prevent them with isolated outputs, controlled concurrency, atomic conditional writes and guarded model promotion—without unnecessarily serializing independent ML work.
What is the difference between pipeline orchestration and data transformation?
Pipeline orchestration coordinates when, where, and in what order tasks execute . Data transformation changes the contents or structure of data . They work together, but they solve different problems. An orchestrator can run a transformation; the transformation logic can be executed without that orchestrator. Beginner friendly example: preparing delivery time data A food delivery company has raw order records: A data transformation performs actual computations: 1. Remove cancelled orders. 2. Convert distance m to distance km by dividing by 1,000 . 3. Calculate average delivery time for the retained orders. The resulting rows have distances 2.5 km and 1.8 km . Their average duration is (20 + 15) / 2 = 17.5 minutes . The transformation changed the data itself. The orchestrator coordinates the larger ML process: It launches the transformation after data extraction, waits for it to complete successfully, schedules training only after validation, and records each task's state. The orchestrator does not decide how to convert meters to kilometers; transformation code does. What does orchestration own? Task order: Decide which operations depend on others. Execution: Launch jobs on the appropriate workers, clusters, or external systems. Triggering: Start runs according to an event, time schedule, or manual request. Failure handling: Apply retry, timeout, and task state rules. Operational controls: Restrict concurrency and resource use. Visibility: Track run IDs, statuses, failures, and links to logs and outputs. Examples include Apache Airflow , Kubeflow Pipelines , Dagster , and Prefect . These responsibilities are about coordinating computation , rather than defining the computation's mathematical result. What does data transformation own? Filtering invalid or unwanted records. Converting units, types, formats, or timestamps. Joining orders with reference tables. Handling missing values and encoding categorical features. Aggregating transactions into features. Producing a well defined output dataset or feature artifact. Tools include SQL, dbt, pandas, Apache Spark , and custom Python components. They do not all operate in the same way: dbt commonly organizes SQL based transformations, while Spark supports distributed data processing. How do the two layers work together? A typical production ML pipeline uses both: 1. Orchestrator: Start a run when an approved data snapshot arrives. 2. Transformation task: Clean and aggregate the snapshot into training features. 3. Orchestrator: Confirm the task succeeded and its output is available. 4. Validation task: Check schema, missing values, and expected feature ranges. 5. Orchestrator: Release training only after validation passes. 6. Training task: Fit the model using the verified feature dataset. The transformation program may run inside an Airflow worker, or the Airflow task may submit a remote Spark/dbt job. Running remotely does not change the distinction: the orchestrator controls the workflow, and the transformation engine processes records. Simple code illustration Here is a small transformation function: This code contains data logic , but no scheduling. An Airflow task can call it within a workflow: This illustrates task orchestration through dependencies. The transformation task body is intentionally simplified; a real pipeline must load a versioned dataset, persist durable artifacts, and avoid passing large dataframes through task metadata channels. What can go wrong if you confuse them? Mistake 1: Put complicated transformation logic into the scheduler definition. Running heavy data processing while the orchestrator parses or schedules workflows can reduce reliability. Keep computational work in task execution or purpose built processing services. Mistake 2: Assume a successful task means the data is correct. The orchestrator may report success while the transformation produced incorrect features. Test transformations and add explicit data quality checks. Mistake 3: Use orchestration as a substitute for versioned datasets. Scheduling a job does not guarantee reproducibility. Record the input snapshot, code version, parameters, and output artifact identity. Mistake 4: Add unnecessary orchestration for a simple one time script. A local transformation can be sufficient for exploratory work. Use orchestration when repeatability, dependencies, automation, recovery, and operational visibility justify it. Which layer should you choose first? If your problem is "How do I compute correct, usable features?" , design and test the transformation. If your problem is "How do I reliably run extraction, feature preparation, training, and evaluation together?" , design orchestration around tested computational units. Question 66 addressed conflicts between simultaneous runs. This lesson instead separates workflow coordination from the data processing operations that workflow tasks execute . In short: Orchestration manages the workflow ; transformation changes the data . A well designed ML pipeline combines both without confusing scheduling success with data correctness.
When should you use Apache Airflow for an ML workflow?
Use Apache Airflow when your ML workflow is a multi step, finite process that must run reliably across data, compute, and model systems. Airflow is particularly effective for batch oriented workflows requiring scheduling, dependencies, integrations, retries, visibility, and historical reruns. It is not a model training framework : Airflow coordinates training jobs but PyTorch, scikit learn, Spark, SageMaker, or another system does the actual computation. Question 67 explained the general distinction between orchestration and transformation. This lesson is about whether Airflow is the right orchestrator for a particular ML workload . Beginner friendly example: daily delivery time predictions A food delivery company retrains its delivery time model from the previous day's validated orders: Suppose the steps take 3, 12, 30, 5, and 2 minutes respectively. When they must run sequentially, the critical processing time is 52 minutes (excluding queue and startup delays). Airflow can create the run, coordinate each step, retry a transient failure, and show which task did not complete. The 30 minute training job should usually run on appropriate compute, such as a separate container, GPU worker, or managed training service. Airflow manages its execution and result; it need not perform GPU training inside the scheduler. Six signals that Airflow is a good fit 1. Your workflows are finite and repeatable. A training pipeline starts, performs bounded work, and finishes. Daily retraining, scheduled batch inference, feature generation, and periodic evaluation are good examples. 2. Several systems must be coordinated. An ML run might use object storage, a warehouse, Spark, a Kubernetes job, a managed model training service, and a model registry. Airflow operators and provider integrations connect these tasks. 3. You need mature operational scheduling. Airflow supports scheduled and event initiated batch runs, dependencies, catchup/backfills, retry controls, concurrency limits, and a UI for examining run history. 4. Your team works with Python and version control. Workflows are Python code that can be reviewed, tested, deployed, and maintained through normal software engineering practices. 5. Failures need clear ownership. Operators can inspect a task's status and logs, fix a problem, and rerun affected work without blindly restarting every previous step. Side effects must still be designed safely. 6. The organization already operates Airflow. Reusing an established platform can be simpler than introducing another orchestrator just to coordinate a few ML specific tasks. What are Airflow's strengths? Broad integrations: Providers connect to cloud infrastructure, databases, storage, containers, and processing services. Python workflow definitions: Easy to represent dependencies, parameterization, task groups, and branching. Operational tooling: DAG runs, task states, logs, failure alerts, retries, backfills, and concurrency controls. Flexible execution: Can submit external compute rather than centralizing all work on Airflow workers. Extensible workflows: Dynamic task mapping can generate work for a variable set of datasets or partitions. Airflow's value is the coordination of a larger production system , not acceleration of model training itself. Where is Airflow a poor fit? Continuous real time inference or unbounded streaming: Airflow's core design is finite, batch oriented workflows. It can orchestrate setup and batch follow up jobs, but a streaming engine or serving platform is normally responsible for continuously processing events or responding to requests. Tightly coupled GPU native pipelines: If the main problem is orchestrating GPU heavy training components with native ML artifact management and Kubernetes focused execution, evaluate Kubeflow Pipelines or a training platform rather than assuming Airflow is always the best choice. Tiny one off experiments: A notebook or straightforward Python script may be simpler than operating a scheduler. Teams that need low code authoring: Airflow emphasizes workflows as code. Its UI is strong for monitoring and operating runs, but writing DAGs generally requires Python. Expecting built in experiment tracking: Airflow logs task execution, but model metrics, experiment comparisons, data lineage, and feature metadata typically need integrations with specialized systems. Airflow does not replace MLflow, a registry, or data quality tooling. A small Airflow 3 style code example This example is a dependency illustration , not a complete ML training implementation. In real workloads, tasks should exchange small references to versioned datasets/models, not large model objects through Airflow metadata. An evaluate result of True is not a production deployment approval. A quick selection checklist Ask these questions before choosing Airflow: 1. Is the workflow batch oriented and bounded , not a continuously running stream? 2. Does it coordinate several dependent tasks or external services ? 3. Does the team need schedules, retries, historical runs, and operational visibility ? 4. Can heavy ML computation execute in external systems with clear resource management? 5. Is the team comfortable maintaining Python DAGs and Airflow infrastructure , or does it have a managed Airflow service? 6. Can model metrics, artifact lineage, evaluation, and deployment approval be handled by appropriate companion tools? When most answers are yes, Airflow is a strong candidate. When the workload is mainly interactive experimentation, specialized Kubernetes native training, or low latency serving, compare other tools. Practical decision A team that already has Airflow coordinating nightly warehouse jobs and now wants daily feature preparation, managed training, and evaluation may benefit from extending the existing Airflow platform. A team whose primary goal is native component level ML artifact lineage and GPU heavy Kubernetes training should also assess Kubeflow Pipelines (Question 69). The choice is about ecosystem, operational constraints, and workload fit—not a universal winner. In short: Choose Airflow for reliable cross system, finite ML workflows that need Python defined automation and operational controls. Keep expensive training on suitable compute, and use dedicated tools for experiment tracking, model serving, and specialized ML metadata.
When should you use Kubeflow Pipelines for model training orchestration?
Use Kubeflow Pipelines (KFP) when you want reusable, container based ML workflows that run on Kubernetes and treat datasets, models, and metrics as first class artifacts. It is especially useful for teams already operating Kubernetes and repeatedly training, evaluating, and comparing models across scalable compute resources. Question 68 focused on when Apache Airflow is a good fit for broad batch workflow orchestration . This lesson focuses instead on KFP's ML native component model, Kubernetes execution, artifact lineage, and selection tradeoffs . Beginner friendly example: a delivery time prediction model A food delivery company wants a reproducible training workflow: Suppose the dataset preparation takes 8 minutes , GPU training takes 22 minutes , and evaluation takes 4 minutes . With sequential dependencies, the estimated task processing time is 34 minutes , excluding pod startup, image downloads, and resource queueing. KFP lets the team define each operation as a component with inputs and outputs. At execution, the KFP backend launches Kubernetes based work for those components, coordinates data passing, and records run and artifact metadata. For the training component, developers can request GPU resources when the workload needs them. The cluster must have actual GPU capable nodes, usable drivers/device plugins, quota, and scheduling capacity: specifying a GPU request does not create GPU hardware . What makes KFP useful for model training? 1. Reusable ML components. Define preprocessing, training, and evaluation as independent components with declared inputs and outputs. Teams can compose them into different experiments without rewriting the whole workflow. 2. First class ML artifacts. Instead of passing a large in memory model between tasks, KFP can track typed dataset and model artifacts and their storage paths. This helps trace which input dataset and training execution produced a candidate. 3. Kubernetes resource controls. Tasks execute in containers and can request CPU, memory, GPUs, or suitable nodes. Different tasks may have different compute needs. 4. Reproducible workflow definitions. Author in Python using the KFP SDK; compile to an intermediate representation (IR) YAML and submit to a compatible backend. Pin images, inputs, and parameters for meaningful reproducibility. 5. Experimentation and iteration. KFP tracks pipeline runs, parameters, task executions, and artifacts. Component caching can avoid repeating suitable unchanged computations. 6. Parallelism and conditional execution. Independent components can execute concurrently, subject to dependencies and available cluster capacity. Evaluation results can control downstream candidate handling branches. A small practical KFP example This snippet demonstrates Python components, parameter passing, dependencies, and a GPU request . In a real pipeline, store large datasets/models as typed KFP artifacts rather than small strings, use compatible training container images, and configure an authorized artifact store. The training and evaluation functions above are placeholders: they do not actually train or assess a model. Kubernetes prerequisites and operating cost KFP is not simply a Python package that turns a laptop into a scalable ML platform. For its standard open source backend, plan for: Kubernetes environment: An existing cluster with worker nodes, suitable CPU/GPU capacity, namespace permissions, and an appropriate KFP backend deployment. KFP may be installed with Kubeflow or as a standalone backend. Container execution: Images, dependencies, registry access, and security scanning. Components need usable runtime libraries and compatible accelerators. Artifact storage and metadata: An accessible pipeline root/object store, credentials or workload identities, and durable metadata storage. Networking and security: Access to datasets, registries, tracking systems, and model services under least privilege RBAC/IAM rules. Platform operations: Cluster upgrades, autoscaling, quotas, persistent storage, observability, backups, and troubleshooting failed pods. The Python SDK can compile pipeline specifications without you personally operating a cluster, and a managed KFP compatible backend (such as Vertex AI Pipelines) changes who operates the infrastructure. But open source KFP execution still needs a supported backend and compute resources . Kubernetes familiarity is a meaningful operational requirement. When KFP may not be the best choice Small experiments or infrequent jobs. A notebook, script, or managed training job may be faster to maintain than a full pipeline platform. No Kubernetes skills or platform team. Running a cluster and metadata infrastructure purely for a few monthly training tasks may be excessive. Mainly data warehouse and business process orchestration. If most tasks coordinate SQL warehouses, APIs, reporting, and dozens of non ML systems, the broader provider and operations ecosystem of Airflow may be more natural. Continuous low latency serving. KFP is designed for workflow executions, not as a replacement for real time model serving infrastructure. Assuming artifact tracking solves everything. KFP metadata and artifact lineage do not automatically supply model approval governance, production monitoring, a feature store, or complete experiment management policy. Integrate other systems where needed. How to choose KFP KFP is a strong candidate when: 1. ML workflows contain multiple reusable training and evaluation components . 2. The organization already has Kubernetes or a suitable managed KFP backend . 3. Tasks need different resource profiles , including GPU training. 4. The team values typed ML artifacts, lineage, versioned runs, and iteration . 5. Platform engineering can manage storage, container security, identity, and cluster operations . 6. The workload is a finite pipeline , rather than an always on inference service. If those do not describe your setup, the simplest reliable platform may be a managed training workflow or an orchestrator you already operate. A useful design principle KFP makes it convenient to connect training steps, but correctness still belongs to the components . Validate datasets, pin their versions, track model parameters, use intentional caching, and keep production promotion behind evaluation and authorization gates. In short: Choose Kubeflow Pipelines for Kubernetes backed, ML first orchestration with reusable container components, scalable execution, and model/dataset artifact tracking. Account for infrastructure and team expertise before adopting it; use complementary tools for serving, specialized tracking, and production governance.
How do Prefect and Dagster approach workflow orchestration?
Prefect and Dagster both orchestrate Python based workflows, but they organize the work differently. Prefect is primarily flow first : define functions and control their execution. Dagster is strongly asset first : define the datasets, models, or other assets your system produces and how they depend on one another. Both can coordinate machine learning pipelines. Neither automatically trains a model or replaces a model registry. Beginner friendly example A food delivery company wants to build a model that predicts delivery times: 1. Load yesterday's approved order snapshot. 2. Clean the orders and generate features. 3. Train a model. 4. Evaluate and save the candidate. Imagine feature preparation takes 5 minutes , model training 15 minutes , and evaluation 2 minutes . The three steps require about 22 minutes of task processing when executed sequentially, excluding startup and queue time. In Prefect , the developer normally starts with a Python flow that calls tasks : “Clean this snapshot, train a model using the resulting features, then evaluate it.” In Dagster , the developer can start with named assets: “These validated features come from this order snapshot; this model comes from these features; these evaluation results belong to this model.” Both can execute the same work. What changes is the programming model and the way work and outputs are represented . How does Prefect approach orchestration? Prefect uses Python decorated flows and tasks . The developer expresses workflow logic using familiar functions and control structures, while Prefect supplies orchestration state, retries, deployments, scheduling, observability, and execution infrastructure. A simplified example: Why teams like it: Python oriented authoring is approachable for ML engineers. Control flow can be expressed through regular Python rather than an entirely separate graph specification. Deployments, work pools/workers, retries, concurrency limits, and UI visibility support production operations. It can launch work on varied infrastructure through integrations and execution options. Tradeoffs: You still design durable artifact storage, model lineage, data contracts, security, and promotion controls. A flow first definition alone does not give you a fully managed feature store or asset catalog. Production deployment requires an appropriate Prefect server/cloud and execution setup, depending on the chosen mode. How does Dagster approach orchestration? Dagster emphasizes software defined assets : named pieces of data or other outputs whose dependencies, materializations, metadata, checks, and freshness matter. That fits many ML pipelines in which a validated dataset, feature table, trained model, and evaluation result should remain understandable over time. A simplified example: Dagster infers the dependency from the prepared features input. The produced asset names and materialization history help operators understand what has been created, when, and from which upstream assets. Why teams like it: First class asset lineage helps connect datasets, features, and models. Asset checks and freshness policies help protect downstream consumers. Partitions represent recurring datasets or date based feature computations. The asset graph provides a practical view of what data exists and what needs updating. Dagster also offers ops and jobs for procedural workflows where asset modeling is not ideal. Tradeoffs: Asset based architecture requires thoughtful definitions for asset identities, partitions, storage, and materialization policies. A team concerned only with orchestrating a few arbitrary API calls may find the asset model more structure than it needs. Dagster still relies on external systems to store large artifacts and run specialized training. When might you choose Prefect? Prefect is often attractive when: The team writes most ML automation in Python and wants to turn existing functions into managed flows. The workflow has dynamic branching or varied external service calls. The primary abstraction is a series of actions to execute . The team values flexible deployment patterns and explicit task/run level orchestration. Example: an ML service must fetch configuration, call a document parser, run different evaluation branches according to input type, notify reviewers, and submit approved models to an external service. When might you choose Dagster? Dagster is often attractive when: The central problem is maintaining datasets, feature assets, model versions, and evaluation outputs . Teams need rich lineage and to understand which upstream artifacts contributed to a model. Data partitions, materialization history, freshness, and checks are important. The organization wants assets to serve as the operational map of its data and ML platform. Example: a team owns dozens of daily feature datasets and model artifacts, each with explicit source dependencies and quality checks. Important limitations and overlap Do not interpret this as “Prefect cannot track artifacts” or “Dagster cannot run dynamic Python workflows.” Both support substantially more than their primary philosophical emphasis. They also both need decisions about workers, resource quotas, credentials, and artifact storage. A short 22 minute sequential pipeline does not automatically become faster because you switch orchestrators. Performance usually depends more on data processing engines, worker availability, model training hardware, and whether independent tasks can run in parallel. Similarly, either system can execute a validation task that passes technically but produces wrong data. Add explicit quality checks, immutable input versions, and safe release gates. A simple decision rule Prefer a flow first perspective when the most important question is “What actions should this Python workflow execute, and when?” Prefer an asset first perspective when the most important question is “What data or model assets should exist, and what upstream work makes them valid?” Evaluate team familiarity, operational hosting, existing integrations, artifact needs, and deployment expectations before making the final platform choice. In short: Prefect focuses on orchestrating Python flows and tasks; Dagster puts software defined assets and their lineage at the center. Both can run ML workflows, with different abstractions and operational tradeoffs. The next lesson builds a broader orchestrator selection framework.