How do you coordinate multiple datasets with different update schedules?

Coordinate datasets with different update schedules by assigning each training run one logical cutoff time, then resolving the exact completed version of every required input that is valid for that cutoff. A scheduler should not simply join the newest hourly, daily, and weekly tables: those versions may refer to incompatible points in time and accidentally introduce future information.

Question 111 covered whether a particular snapshot is approved for training, and Question 102 covered detection of missing or delayed arrivals. This lesson focuses on time alignment across multiple independently updated sources, release barriers, cross-source consistency and targeted reprocessing.

Beginner-friendly example: delivery ETA model

Suppose a food-delivery service retrains an ETA model every morning from three upstream sources:

• Hourly traffic: Completed hourly traffic aggregates, normally finalized 10 minutes after each hourly window.
• Daily orders: Prior-day order/label data, finalized at 06:00 UTC the following morning.
• Weekly restaurant profiles: A weekly snapshot published every Monday at 04:00 UTC, representing approved profile values effective from that Monday.

The downstream training DAG starts its readiness check at 07:00 UTC on Tuesday, November 10, 2026. It processes order events from Monday, November 9, 00:00–Tuesday, November 10, 00:00 UTC, using half-open intervals [start, end).

For this training slice it should use:

• 24 hourly traffic partitions covering 2026-11-09 00:00 through 23:00, each committed, with its own hourly interval.
• One complete daily orders partition for 2026-11-09.
• The Monday weekly profile revision effective for 2026-11-09, provided it was genuinely available at the corresponding historical prediction time. If the weekly dataset changes during the day, use effective-time versioning rather than applying future changes to earlier events.

It should not blindly choose the newest traffic partition at 07:00 on Tuesday: that would include Tuesday morning traffic after the Monday training examples and may leak future data into historical features.

Numerical readiness example

At the Tuesday 07:00 readiness check:

• Hourly traffic has 23 of 24 required committed partitions: 23/24 = 95.83% interval coverage.
• Daily orders has 1 of 1 required partition: 100%.
• Weekly profiles has 1 of 1 required effective snapshot: 100%.

If the full 24-hour traffic coverage is mandatory, training must wait despite the other two sources being complete.

Suppose the missing hourly traffic partition becomes committed at 07:18. The readiness barrier then passes, subject to all snapshot-integrity and approval checks.

If company policy instead allows at most one missing traffic hour with a documented fallback, the workflow could proceed only after tagging that hour, imputing the approved fallback and evaluating bias from missing coverage. 95.83% time-interval coverage is not necessarily 95.83% record, traffic-volume or user coverage.

For a weekly dataset covering 7 days, a daily run can reuse one weekly snapshot for each day within its validity interval; it should not force the weekly source to publish daily.

The key concepts: five clocks and IDs

• Event time: When a delivery or traffic observation actually happened.
• Processing time: When the event was processed by a worker.
• Publication/commit time: When an entire source version became available and verified.
• Effective time: The business period during which a weekly restaurant profile or category mapping applies.
• Logical run interval: The historical period a training job intends to process, independent of when it actually executes.

Keep logical_interval_start, logical_interval_end, snapshot_id, valid_from, valid_to, committed_at and source version identifiers in a consistent, documented timezone (often UTC). Do not confuse a weekly file's upload timestamp with the week its values describe.

Airflow's data interval is the interval being processed; the run's logical date normally identifies its start under data-interval timetables. Manual and trigger-based runs may use different timetable semantics, so bind your own intended training cutoff explicitly when necessary.

Build a cross-source alignment contract

For every dependency, specify:

Hourly traffic

• Required interval: every hour intersecting the training window.
• Completion rule: committed hourly snapshots with applicable allowed lateness.
• Join rule: match traffic observations that were available at the original prediction time, not future-derived hourly summaries.
• Gap policy: require 24/24 hours or use a reviewed fallback for sparse hours.

Daily orders

• Required interval: the entire relevant business day.
• Completion rule: published daily snapshot with its validated coverage boundary.
• Join rule: join by order/entity IDs using the correct event time.
• Gap policy: normally treat a missing daily label partition as a mandatory blocker.

Weekly restaurant profile

• Required interval: the appropriate business-effective week or historical version.
• Completion rule: immutable revision with defined valid_from and valid_to.
• Join rule: point-in-time lookup of the profile effective and available when the order was placed.
• Gap policy: use a previous effective version only if explicitly allowed; never substitute a future revision retroactively.

An important distinction: a snapshot can exist today but still be invalid for a historical training row if its features were not known at that row's prediction time. Coordinate both publication history and event/effective-time semantics.

Dependency barrier and consistent input manifest

Training run T with logical interval [Nov 9, Nov 10)
                           |
               Resolve source requirements
                           |
          +----------------+----------------+
          |                |                |
   24 traffic hours     Nov 9 orders    Effective weekly profile
          |                |                |
     Verify commits    Verify commit    Verify effective version
          +----------------+----------------+
                           |
                Cross-source readiness gate
                           |
                  All mandatory inputs ready?
                    /                \
                   No                 Yes
                   |                   |
             Wait / escalate      Freeze version manifest
                                       |
                               Validate aligned joins
                                       |
                                Launch training

Create one immutable multi-input manifest before training, such as:

YAML
training_run: eta-2026-11-09
logical_start: 2026-11-09T00:00:00Z
logical_end: 2026-11-10T00:00:00Z
traffic:
  partition_count: 24
  snapshot_set_id: traffic-hours-20261109-v2
orders:
  snapshot_id: orders-20261109-v1
restaurant_profiles:
  snapshot_id: profiles-week-20261109-v3
  valid_from: 2026-11-09T00:00:00Z
manifest_revision: 1

The IDs are illustrative. In production, resolve immutable snapshot/object-version references, coverage evidence, digests and quality/approval receipts. The manifest as a whole is the reproducible input to a downstream run. Do not allow traffic revisions to change halfway through fitting.

Simple Python multi-source readiness check

PYTHON
from datetime import date, datetime, timedelta, timezone

def expected_hour_starts(day: date):
    start = datetime(day.year, day.month, day.day, tzinfo=timezone.utc)
    return {start + timedelta(hours=i) for i in range(24)}

def ready_for_training(day, traffic_committed, daily_committed,
                       weekly_profile_effective):
    missing_hours = expected_hour_starts(day) - set(traffic_committed)

    problems = []
    if missing_hours:
        problems.append(f"MISSING_TRAFFIC_HOURS:{len(missing_hours)}")
    if day not in daily_committed:
        problems.append("DAILY_ORDERS_NOT_COMMITTED")
    if weekly_profile_effective is None:
        problems.append("WEEKLY_PROFILE_NOT_AVAILABLE")

    return {
        "ready": not problems,
        "problems": problems,
    }

day = date(2026, 11, 9)
hours = sorted(expected_hour_starts(day))
result = ready_for_training(
    day, traffic_committed=hours[:-1],
    daily_committed={day},
    weekly_profile_effective="profiles-week-20261109-v3",
)
print(result)
# {'ready': False, 'problems': ['MISSING_TRAFFIC_HOURS:1']}

This is an educational set-completeness gate, not a secure, production-ready implementation. Real code must check each source's committed state, immutable IDs, quality/approval evidence, timestamps, row-level point-in-time correctness and weekly effective-window semantics. Also handle daylight-saving changes if local business days—not UTC days—define the partitions; local days can contain 23 or 25 hourly intervals.

Watermarks, lateness and cutoff times

Streaming systems use an event-time watermark to estimate progress through their input events. In Apache Beam, a watermark moving beyond a window boundary can trigger a result, but late events may still appear according to allowed-lateness and trigger policies. A watermark is not absolute evidence that all source events exist.

For hourly traffic:

• Hour window: [14:00, 15:00).
• Event-time watermark reaches 15:05, past the window end.
• The pipeline may materialize an initial completed window based on its trigger policy.
• Its agreed late-data horizon might extend to 15:30; corrections received before that cutoff could require a revised hourly snapshot.

The team must choose whether daily training at 07:00 consumes finalized hourly windows or provisional windows with revision tracking. If a provisional snapshot is allowed, save its exact revision and correction policy; never silently swap it later.

Use separate cutoff time policies:

• When is a source considered sufficiently complete for a daily training run?
• How long should the orchestrator wait for corrections?
• At what point is a late correction assigned to a controlled backfill instead of modifying the current run?

Prefer a business-aligned cutoff that reflects expected labels and feature availability, not merely the first moment every API responds.

Scheduling and orchestration

An orchestrator such as Airflow can run hourly producer DAGs, daily orders jobs and weekly profile updates under independent timetables. A downstream training DAG should depend on specific logical data intervals, not simply on whether the latest producer DAG succeeded.

A robust approach:

1. Publish completed source versions and partition identities to a trusted catalog/event source.
2. At each daily training interval, compute the exact hourly/daily/weekly dependencies.
3. Wait for the dependency manifest using sensors, dataset/asset events or a catalog query; apply a bounded timeout.
4. Validate the aligned version set, then publish the immutable multi-input manifest.
5. Pass that manifest ID to training, rather than repeatedly reading source aliases like latest.
6. Monitor dependency readiness and produce alerts for a stalled run. The missing-arrival incident mechanics were covered in Question 102.

Airflow's timetable/data-interval semantics determine which period a run processes. For unusually complex calendars, use a timetable or explicit logical-interval mapping. Confirm the Airflow version's behavior before relying on a particular trigger/logical-date association.

Partial availability and graceful degradation

Not all inputs deserve equal treatment:

• Mandatory labels or entity mapping: Block if incomplete, because the training target or join becomes unreliable.
• Optional external context: Permit a documented fallback if coverage and model impact are assessed.
• Weekly reference data: Reuse the latest *historically valid* prior effective revision when the source contract permits it.
• Provisional hourly signals: Use only if the training methodology explicitly supports snapshots that can later be corrected.

Record source coverage alongside trained-model metrics. A seemingly good score may hide excluded orders from missing traffic hours.

In the example, skipping one missing hour in a peak period could eliminate disproportionately many delivery orders. The simple 23/24 coverage fraction cannot establish representativeness.

Backfills across mixed cadences

Suppose traffic for November 9 at 23:00 is corrected after a daily ETA training run has already been completed.

• Publish a new immutable traffic partition revision.
• Identify which daily training/feature windows depended on that hour.
• Determine whether the new revision changes model features materially or violates a correction policy.
• Trigger a targeted backfill only for affected daily intervals and downstream artifacts.
• Pin the revised traffic partition together with the *same intended historical* daily and weekly snapshot versions, unless those versions were also explicitly revised.
• Use an idempotency key such as eta:2026-11-09:manifest-v2.
• Keep the original run's manifest and record why its successor was created.

If a weekly reference dataset is corrected retroactively, a single revision can affect up to seven daily training intervals (or more when its effective period is different). Do not rerun all historical data without calculating the actual impacted dependency intervals.

Be careful with latest inputs during replays: a backfill started months later must reproduce the original logical window, not leak current profiles or future labels into the past.

Trade-offs

• Wait for full cross-source consistency: More reliable but delays training when one source lags.
• Use provisional windows: Faster updates, but increases revisions and potential backfills.
• Reuse prior weekly data: Stabilizes scheduling, but may miss real profile changes.
• Freeze snapshots: Reproducible experiments, with extra metadata/version-storage costs.
• Recompute every affected day: Highest historical consistency, potentially expensive.
• Selective backfill: Saves compute, but requires trustworthy dependency lineage.
• Use one global cutoff for all sources: Simpler to explain, though sometimes overly strict for sources with different valid-time semantics.

Mistakes to avoid

• Joining latest hourly traffic to yesterday's training labels.
• Assuming three individually ready datasets are automatically temporally compatible.
• Treating a streaming watermark as proof of perfect completeness.
• Ignoring effective dates for weekly reference tables.
• Assuming every business day always contains exactly 24 local hours.
• Using current restaurant profiles for historical orders when those profiles changed.
• Overwriting previously selected source revisions during a running training job.
• Skipping the cross-source manifest and losing the exact inputs needed to reproduce a model.
• Triggering indiscriminate backfills when only one partition changed.
• Treating a partially available high-volume hour as harmless because 23 of 24 hour partitions passed.

In short: Give each training run a logical interval, align hourly, daily and weekly sources by event/effective time, wait at a defined multi-input readiness barrier, and freeze one consistent manifest of immutable source versions. Reconcile late corrections through targeted, auditable backfills rather than silently changing historical inputs.