How do you build pipeline-level observability and run histories?

Pipeline-level observability means being able to answer: *What ran, when, with which inputs, what happened in every step, and which artifacts were produced?* A run history preserves that evidence across executions, not just the latest status.

This is different from Question 64: there we chose how to recover a failed workflow; here we build the records and signals needed to understand and compare runs, whether successful or failed.

Practical example: delivery-time training

A food-delivery company trains a model daily. Its orchestrator assigns a stable identifier to each workflow execution:

pipeline: delivery-time-training
run_id: delivery-train-2026-10-26-001
trigger: scheduled
dataset_snapshot: orders-2026-10-25-v2
pipeline_version: git:8c20fa1
model_image: sha256:abc123... (illustrative)
started_at: 2026-10-26T02:00:00Z
status: FAILED

The run contains these step records:

• validate_orders: SUCCESS — 30 seconds — 120,000 rows accepted.
• build_features: SUCCESS — 3 minutes — artifact features/orders-2026-10-25-v2.
• train_model: SUCCESS — 12 minutes — artifact models/candidate-842.
• evaluate_model: FAILED — 1 minute — error: evaluation reference dataset unavailable.
• register_model: NOT_STARTED — no model registered.

Someone opening this run should see the failed evaluation task, its timestamped logs, its missing input, and the affected downstream step. They should *not* have to search several worker machines to discover which training data was used.

What should you capture?

1. Pipeline run identity and context

Record pipeline_id, unique run_id, start/end timestamps, trigger type, effective parameters, dataset snapshot IDs, environment, pipeline definition version, and image digest. Use one run_id in logs, metrics, tracing, and artifact metadata.

2. Step execution history

For each task capture task_id, attempt number, state (queued, running, success, failed, skipped), start/end times, duration, worker identity where appropriate, error category, and link to relevant logs. Distinguish a task *waiting in queue* from one actively consuming compute.

3. Structured logs

Logs should answer what each component tried to do and why it failed. Example:

JSON
{
  "run_id": "delivery-train-2026-10-26-001",
  "task_id": "evaluate_model",
  "attempt": 1,
  "level": "ERROR",
  "event": "evaluation_dataset_missing",
  "snapshot_id": "orders-2026-10-25-v2"
}

Include correlation IDs and meaningful errors, but do not log credentials, customer records, or sensitive training examples.

4. Operational and ML metrics

Operational metrics include total runtime, per-step duration, queue delay, task failures, retry counts, CPU/GPU utilization, and successful-run percentage. Model metrics can include MAE, RMSE, F1, or another evaluation measure appropriate to the task, alongside the evaluation dataset version.

5. Artifact lineage

Connect input dataset → feature artifact → trained model → evaluation artifact. Each artifact should have a durable identity, version or digest, storage location, producing run_id and task_id, and relevant parent artifact IDs. This makes it possible to determine *which data and code produced a particular model*.

How does run history help? A numerical comparison

Three recent runs of the same delivery-time pipeline have durations:

• Run A: 16 minutes, SUCCESS
• Run B: 18 minutes, SUCCESS
• Run C: 31 minutes, FAILED during evaluation

The historical average elapsed time is (16 + 18 + 31) / 3 = 21.7 minutes (including the failed run). The success rate is 2/3 = 66.7% across this tiny sample.

These figures flag something to inspect, but three runs are too few to establish a reliable baseline. Per-task durations, queue delays, and logs can show whether Run C spent extra time waiting for GPUs, training, or failing an external lookup.

How do tools support this?

Apache Airflow: The DAG Runs and Grid views show historical runs, task states, attempts, timing, and task logs. They are useful for operational diagnosis and comparing multiple pipeline executions. Add external logging/metrics retention when task workers are ephemeral.

Kubeflow Pipelines: Tracks pipeline runs, component executions, and ML artifacts. Its ML Metadata store records execution and artifact properties and can visualize lineage across runs, including an artifact consumed by more than one run.

ML experiment trackers: Systems such as MLflow can complement an orchestrator by recording hyperparameters, evaluation metrics, and model artifacts. Use a shared run correlation ID to join experiment records with pipeline run history.

A practical observability checklist

1. Emit the same run ID from every task and store it in all artifacts.
2. Record exact input snapshots, resolved parameters, component code/image versions, and output checksums.
3. Capture task states and attempt counts, including skipped and failed steps.
4. Centralize searchable structured logs, retaining useful error details without secrets.
5. Monitor queue times, runtimes, failures, and data-quality or model-evaluation metrics.
6. Maintain dataset-to-model artifact lineage and retain run history according to governance needs.
7. Alert when expected runs are missing, critical tasks fail, outputs are late, or model-quality gates are violated.
8. Test the workflow by locating one model artifact and tracing it back to its producing run and source snapshot.

In short: Pipeline observability combines run IDs, step states, logs, metrics, and artifact lineage so teams can diagnose failures, explain model provenance, measure reliability, and compare executions over time.