How do task dependencies determine pipeline execution order?

Task dependencies determine which ML pipeline steps may start and which must wait. An orchestrator checks each step's upstream requirements and only schedules it when its configured conditions are met.

The previous lessons introduced pipeline artifacts (Question 53) and the DAG structure (Question 54). Here the focus is runtime scheduling: when tasks become eligible, what can run concurrently, and what happens if a prerequisite fails.

A beginner-friendly example with numbers

A food-delivery company trains a model for predicting delivery times. Assume five tasks and their estimated durations:

1. Extract data (A): 2 minutes.
2. Validate data (B): 3 minutes; depends on A.
3. Train model (C): 8 minutes; depends on B.
4. Calculate baseline metrics (D): 4 minutes; depends on B.
5. Evaluate candidate against baseline (E): 2 minutes; depends on both C and D.

The dependency relations are:

A (2 min) → B (3 min) ┬→ C (8 min) ─┐
                      └→ D (4 min) ─┴→ E (2 min)

If workers are available and the duration estimates hold:

• Minute 0–2: A runs. All others wait.
• Minute 2–5: B runs after A succeeds.
• Minute 5–13: C runs after B succeeds.
• Minute 5–9: D also runs after B succeeds; it can run in parallel with C.
• Minute 13–15: E runs because both C and D are complete.

Total time: 15 minutes with enough worker capacity. If the tasks ran strictly one after another, the total would be 19 minutes (2+3+8+4+2). The four-minute difference comes from allowing the independent C and D branches to overlap.

The critical path is A → B → C → E (15 minutes); speeding up D alone would not make the entire pipeline finish sooner while C remains the longer prerequisite.

What makes a task ready to run?

An orchestrator generally considers:

• Upstream completion: Have the required predecessor tasks finished in the appropriate states?
• Trigger rule: Does the task require all predecessors to succeed, or a different condition?
• Capacity: Is a worker slot or appropriate compute resource available?
• Other scheduling controls: Do pool limits, concurrency quotas, scheduling windows, or task state restrictions permit the run?

A task can therefore be eligible according to dependencies but still remain queued while waiting for a worker. Dependencies determine valid order; resource availability determines actual start times.

How are dependencies written?

In Apache Airflow, the >> operator can declare upstream-to-downstream relationships:

PYTHON
extract >> validate
validate >> [train, baseline]
[train, baseline] >> evaluate

These lines are a dependency fragment, not a complete runnable DAG: the task objects and DAG must be defined separately.

In Kubeflow Pipelines, passing a task's output into another component's input can establish ordering automatically. For a control dependency without data passing, a task can explicitly depend on another task (for example, with an .after(...) relationship in the relevant DSL).

A useful rule is: Add an edge only when one task really must precede another. A needless dependency forces otherwise independent work to run in sequence.

What happens when a prerequisite fails?

Suppose validation B fails at minute 4. Training C and baseline D should not start with invalid data. Under a typical all-upstream-success rule, downstream evaluation E also cannot run successfully.

Some pipelines intentionally configure exceptions:

• An alerting task may run after failure.
• A cleanup task may run regardless of whether earlier steps succeeded.
• A reporting task may run when one of several optional branches completed.

In Airflow these behaviors are configured using trigger rules, such as all_success or all_done. all_done means upstream tasks finished in terminal states; it does not mean all succeeded. A workflow should choose rules deliberately so that a model is never approved merely because its evaluation task became executable after a failure.

Retries and exponential backoff are separate reliability controls covered in the next concept.

How do you design good execution dependencies?

1. Identify inputs and outputs for each task, then connect genuine prerequisites.
2. Separate independent work so it can run concurrently when resources permit.
3. Set explicit success conditions for quality-critical steps, especially model evaluation and promotion.
4. Test failure cases to confirm that unsafe downstream steps are blocked.
5. Monitor queue time separately from processing time; slow pipelines may be waiting for resources, not running slowly.

In short: Task dependencies create a partial order, not necessarily one fixed sequential list. They prevent tasks from starting before their prerequisites, expose opportunities for parallelism, and combine with trigger rules and available workers to determine actual execution.