How do you schedule recurring ML pipeline runs?

Recurring ML pipeline scheduling means configuring an orchestrator to create new pipeline executions at a planned cadence—such as every night, every Monday, or once per month—without manually starting each run.

A schedule answers when a run should be created. The pipeline definition answers what work the run performs. Reliable scheduling also requires deciding which data period a run processes, what happens when yesterday's run is still active, and whether missed runs should be replayed.

A beginner-friendly practical example

A food-delivery company trains a delivery-time prediction model every morning after the previous day's orders have arrived.

Its requirement is: Run at 02:00 UTC every day, process the previous complete UTC calendar day, and avoid overlapping training jobs.

For instance, the scheduled execution at 02:00 UTC on October 26 should process orders from October 25, 00:00 UTC up to (but not including) October 26, 00:00 UTC. This is a business rule the training code implements; a 02:00 schedule alone does not automatically choose the correct input partition.

The five-field cron expression is:

0 2 * * *
│ │ │ │ └── day of week: every day
│ │ │ └──── month: every month
│ │ └────── day of month: every day
│ └──────── hour: 02
└────────── minute: 00

Important: Cron is interpreted in the scheduler's configured timezone. Using UTC avoids daylight-saving changes, but the business may require an explicit local timezone instead.

A practical Apache Airflow example

This minimal Airflow 3-style example uses a trigger timetable so the intent is easy to see: run daily at 02:00 UTC. The task deliberately derives yesterday's partition from the scheduler-supplied logical date rather than from the machine's changing wall clock.

PYTHON
from datetime import datetime, timedelta, timezone
from airflow.sdk import dag, task
from airflow.timetables.trigger import CronTriggerTimetable

@dag(
    dag_id="delivery_time_daily_training",
    schedule=CronTriggerTimetable("0 2 * * *", timezone="UTC"),
    start_date=datetime(2026, 10, 1, tzinfo=timezone.utc),
    catchup=False,
    max_active_runs=1,
)
def delivery_training():
    @task
    def train_for_partition(**context):
        run_date = context["logical_date"].date()
        partition_date = run_date - timedelta(days=1)
        # Load immutable order snapshot for partition_date,
        # validate it, train/evaluate, and record the run.
        print(f"Training for orders date: {partition_date}")

    train_for_partition()

delivery_training()

This is a teaching example, not a fully implemented training pipeline. The actual training step must validate that the dataset for the selected date is complete and enforce a safe model-promotion policy.

With a trigger timetable, logical_date represents the scheduled trigger point. Other Airflow timetables use data intervals and can assign a different meaning to logical_date. For production code, use the selected timetable's documented logical-date/data-interval semantics rather than assuming every timetable is interchangeable.

Four scheduling decisions that matter

1. Cadence: how often?

Choose daily, hourly, weekly, or monthly based on how quickly the data or model changes and what operations can support. A high-frequency schedule with an hour-long training job may create unnecessary pressure. Model updates do not have to be published after every scheduled training run.

2. Timezone and input data interval: what time and which records?

Define both the schedule timezone and the data window. A run starting at 02:00 should not accidentally train on incomplete data for the current day. If data is delivered late, use an explicit completeness check or delay until the source data is ready.

3. Overlap: what if yesterday's run is still running?

Airflow's max_active_runs=1 limits concurrent DAG runs. In Kubeflow Pipelines, a recurring run configuration can also limit concurrent runs. This protects expensive training infrastructure and shared outputs, though the orchestrator may queue later work. Decide whether delayed runs should wait, be skipped, or be handled manually.

4. Catchup and missed schedules: what if the scheduler was paused?

With catchup=False, Airflow normally avoids automatically creating every historical scheduled run since the start date. With catchup enabled, older eligible intervals can generate runs, potentially leading to many costly training executions. Backfills for earlier dates should be a conscious decision with pinned input snapshots; they are covered in a later lesson.

How is recurring scheduling configured in Kubeflow Pipelines?

Kubeflow Pipelines supports recurring runs with either a cron trigger or a periodic interval. A recurring run stores a repeatable pipeline/run configuration and creates executions according to the trigger. Teams can enable or disable a recurring run and configure how many copies may run concurrently.

This is similar in intent to an Airflow DAG schedule, though the APIs and execution semantics differ.

What should be monitored?

• Expected vs. created runs: Was today's run actually scheduled?
• Start delay: Did it wait for a worker or resource quota?
• Completion status and duration: Did the run finish before its operational deadline?
• Data completeness: Did it process the intended snapshot or partition?
• Overlaps and skipped/missed dates: Did concurrency controls or a paused schedule leave gaps?
• Release outcome: Was a candidate only trained, or actually approved and deployed?

A successful scheduler tick is not proof that training completed or a model was safe to deploy.

Scheduling versus the previous lesson

Question 60 covered how one pipeline accepts different dataset IDs, environments, and hyperparameters. Here those inputs can remain fixed or be resolved for each scheduled date; the key new topic is automatic run creation, calendar semantics, time zones, catchup, and concurrency.

The next question compares recurring time-triggered runs with pipelines started by events, such as an incoming dataset.

In short: Schedule ML pipelines with an explicit cadence and timezone, make the processed data window unambiguous, bound concurrency, handle missed intervals deliberately, and monitor each run from scheduling through completion.