How do you prevent stale or unapproved datasets from entering training?

Prevent stale or unapproved datasets from entering training by enforcing a dataset-admission gate before any training job reads its inputs. The gate must resolve an immutable dataset snapshot, verify its approval and quality evidence, check the dataset is recent enough for this specific training use case, and record what it admitted. If required evidence is missing, fail closed.

Question 110 evaluated a *trained model* before registry approval. This lesson evaluates the *training input* before model fitting. Earlier lessons explained online feature freshness (Question 94), upstream arrival delays (Question 102), and dataset quality tests (Question 101); the new concern is combining those signals into a binding, auditable admission decision that cannot be bypassed by switching to a different dataset path after approval.

Beginner-friendly example: delivery ETA retraining

A food-delivery team retrains an ETA model each morning at 08:00 UTC, using the most recent approved daily orders snapshot. Its policy requires:

• Dataset's latest included complete order date is at most 1 calendar day behind the required reporting date.
• Dataset status is exactly APPROVED; PENDING, REJECTED, REVOKED and missing status do not qualify.
• Quality evidence was recorded as PASS for the same immutable snapshot ID, with a supported validation-policy version.
• The dataset came from an authorized, committed source and its checksum/manifest matches what the gate approves.
• Required legal, access and data-usage approvals are valid for the planned training use.

At 08:00 on November 6, the expected reporting date is November 5 (UTC). Three snapshots are available:

• Snapshot A: through November 5, approved, quality PASS, digest verified → ADMIT.
• Snapshot B: through November 5, quality PASS, but status PENDING → REJECT until approved.
• Snapshot C: through November 3, approved, quality PASS → REJECT as too old for this specific policy.

The gate selects A by its version and digest, never by a mutable path such as orders/latest.

Numerical example: freshness and approval are independent

Suppose the gate uses an elapsed-time bound of 36 hours for the newest observation in the approved training snapshot.

• Training starts on November 6 at 08:00 UTC.
• Snapshot A's coverage watermark is November 5 at 12:00 UTC → age 20 hours → freshness PASS.
• Snapshot C's coverage watermark is November 4 at 12:00 UTC → age 44 hours → freshness FAIL by 8 hours.

Even a recent snapshot with a fresh 20-hour watermark must be refused if its approval status is PENDING. Conversely, a previously approved snapshot must not be assumed fresh forever.

Important: A coverage watermark represents the most recent included business event or complete period; a created_at timestamp only shows when someone built the files. Re-exporting old data today does not make the underlying observations current. Define the clock and boundaries explicitly; an event-time maximum by itself does not prove coverage completeness.

Design a dataset admission manifest

Store a signed or access-controlled admission manifest alongside immutable dataset revisions. For example:

YAML
dataset_id: delivery_orders
snapshot_id: orders-20261105-v3
storage_uri: s3://example-training-data/orders/20261105/v3/
content_digest: sha256:example-digest-placeholder
commit_status: COMPLETE
coverage_through_utc: "2026-11-05T12:00:00Z"
quality:
  status: PASS
  validation_run_id: quality-run-981
  policy_version: orders-quality-v4
approval:
  status: APPROVED
  approval_id: data-approval-450
  approved_for: ["eta-model-training"]
  expires_at_utc: "2026-11-07T00:00:00Z"
lineage:
  source_snapshot_id: warehouse-orders-snapshot-772
  transformation_commit: abc123

These are illustrative fields, not a vendor-specific manifest schema. In production, keep approval records in a trusted service rather than trusting arbitrary user-editable metadata in a file.

A stable ID identifies the logical version; a content checksum or immutable object-version reference binds that ID to specific bytes. Store and verify both.

Admission workflow

1. Resolve the intended input: The pipeline requests delivery_orders for a defined logical training date, policy version and purpose.
2. Discover candidates: Query a trusted dataset registry/catalog for committed immutable versions.
3. Validate the source and snapshot: Check object/table version, manifest integrity, authorized location, digest and expected coverage.
4. Verify approval: Resolve a current, authoritative approval record whose scope includes the dataset version and training purpose.
5. Verify quality evidence: Require a passing quality report for the exact snapshot and the supported quality policy; reject unknown or expired evidence.
6. Apply freshness cutoff: Compare the coverage/event-time marker with the training run's fixed logical evaluation time and permitted maximum age.
7. Issue an admission decision: Return the exact version, digest, policy revision, approval and evidence IDs.
8. Bind the job to the decision: Pass an immutable URI/object version or snapshot ID to training; avoid a second lookup of latest.
9. Recheck at use time where necessary: Approval revocation, credential expiry or authorization changes may occur between gate and read.
10. Audit the run: Persist accepted or rejected candidate IDs, reasons, timestamps and actual consumed snapshot.

The result should be a reproducible dataset receipt associated with the training run, not a boolean alone.

A simple Python admission guard

PYTHON
from datetime import datetime, timezone, timedelta

def admit_dataset(meta: dict, now: datetime, max_age_hours=36):
    reasons = []

    if meta.get("approval_status") != "APPROVED":
        reasons.append("NOT_APPROVED")
    if meta.get("commit_status") != "COMPLETE":
        reasons.append("NOT_COMMITTED")
    if meta.get("quality_status") != "PASS":
        reasons.append("QUALITY_NOT_PASSED")
    if not meta.get("snapshot_id") or not meta.get("content_digest"):
        reasons.append("MISSING_SNAPSHOT_IDENTITY")

    coverage = meta.get("coverage_through")
    if not isinstance(coverage, datetime) or coverage.tzinfo is None:
        reasons.append("COVERAGE_TIMESTAMP_UNKNOWN")
    elif coverage > now or (now - coverage) > timedelta(hours=max_age_hours):
        reasons.append("DATA_TOO_OLD_OR_FUTURE")

    expiry = meta.get("approval_expires_at")
    if not isinstance(expiry, datetime) or expiry.tzinfo is None or expiry <= now:
        reasons.append("APPROVAL_EXPIRED_OR_UNKNOWN")

    return {"admitted": len(reasons) == 0, "reasons": reasons}

now = datetime(2026, 11, 6, 8, 0, tzinfo=timezone.utc)
meta = {
    "approval_status": "APPROVED",
    "commit_status": "COMPLETE",
    "quality_status": "PASS",
    "snapshot_id": "orders-20261105-v3",
    "content_digest": "sha256:verified-in-real-system",
    "coverage_through": datetime(2026, 11, 5, 12, 0, tzinfo=timezone.utc),
    "approval_expires_at": datetime(2026, 11, 7, tzinfo=timezone.utc),
}
print(admit_dataset(meta, now))  # {'admitted': True, 'reasons': []}

This is a policy illustration only: it does not cryptographically verify a digest, validate the actual permission scope or bind a quality report to the same snapshot. Production admission must fetch trusted approvals, quality reports and immutable storage identities from controlled services. Never permit a user to manufacture APPROVED by editing a JSON or YAML file.

How an orchestrated pipeline enforces it

Request training at logical time T
             ↓
Resolve immutable dataset candidates
             ↓
Check commit + content identity
             ↓
Check quality evidence + approval scope
             ↓
Check permitted dataset age
             ↓
          ADMIT?
       /          \
     NO            YES
      ↓             ↓
Block + alert    Issue signed/controlled receipt
      ↓             ↓
Do not train     Load exact approved snapshot
                    ↓
             Record consumed version
                    ↓
                 Train model

For an Airflow, Dagster or similar workflow, place this admission step as a hard dependency immediately before data loading. Only emit the immutable dataset identifier after successful checks. Make the data loader reject mutable aliases and unsupported versions independently; a skipped gate should not provide an easy path around policy.

Dagster's asset checks can monitor quality/freshness and prevent downstream asset materialization under the configured execution policy. But a green check still must be associated with the specific version consumed by the training job. Do not treat a green dashboard for yesterday's partition as approval for a different revision published today.

Prevent time-of-check/time-of-use (TOCTOU) bugs

Consider this dangerous sequence:

• At 08:00, orders/latest points to approved snapshot A and the gate passes.
• At 08:01, an upstream process repoints orders/latest to unapproved snapshot B.
• At 08:02, the trainer resolves orders/latest again and reads B.

The gate passed, but training consumed the wrong data.

Fix: The admission result includes immutable snapshot_id=A and the expected content digest; the training loader resolves that exact immutable version and verifies it. If object immutability cannot be guaranteed, use the storage system's snapshot isolation/versioning and validate the actual bytes read against approved identity. The trainer must record the version consumed, not just the version that was checked.

Lineage and allowlists

A training allowlist should be explicit about:

• Dataset names, storage accounts/buckets/tables and allowed source owners.
• Approved snapshot identities and required status.
• Permitted business purposes, environments and model families.
• Approved transformations and feature/label definitions.
• Allowed quality-policy versions and evaluation windows.

A path allowlist alone is insufficient: an allowed bucket can contain an unapproved dataset. Conversely, a correct digest does not prove the source had permission to supply that data. Evaluate identity, integrity, policy approval and authorization separately.

MLflow dataset tracking records dataset name, digest, schema, source and run inputs to support lineage. Log a dataset with its training context and pin the immutable source revision. MLflow tracking is not itself an admission or approval enforcement service, and some dataset types use metadata-only digests: verify the chosen digest's semantics before relying on it for integrity.

Fallback policy: don't secretly use older data

If no eligible snapshot exists:

• Default: Block retraining and continue serving the last validated model.
• Retry: Re-run the admission check after the source is corrected or the required approval arrives.
• Prior approved version: Permit only if a documented exception authorizes a longer age or explicit fallback snapshot, with review of completeness and distribution risks.
• Emergency research use: Optionally allow a clearly labeled nonproduction diagnostic run, but do not treat its output as production-approved or bypass access/data-use constraints.
• Rebuild or backfill: Produce a new version from authorized data, rerun quality and approvals, then request admission normally.

For example, using a two-day-old training snapshot might be acceptable for a low-volatility monthly model but dangerous for a model driven by recent promotions or weather patterns. Even when an older version is approved, it should not silently bypass an age policy.

Auditability and incident investigation

Record:

• Requested dataset and logical training time.
• Candidate versions considered and why each was accepted/rejected.
• Applied admission-policy revision and age thresholds.
• Actual quality report, approval evidence and authorization result.
• Immutable snapshot/digest bound to the data loader.
• Training-run ID, model artifact and downstream registry relationship.
• Any override approver, expiration, justification and nonproduction constraints.
• Alerts for repeated rejection, expired approvals, unknown snapshots and failed integrity checks.

Periodically sample finished training runs and verify their consumed dataset receipts still resolve to valid lineage and the approved snapshot identifiers. Detect mismatches between approved and actually read input versions.

Trade-offs and common mistakes

• Strict freshness: Protects relevance but may delay retraining when data updates are slow.
• Longer freshness windows: Improve availability but can degrade model relevance.
• Snapshot immutability: Improves reproducibility and audit but may increase storage and version-management costs.
• Manual approval: Supports oversight for sensitive datasets but can become a bottleneck; automate routine checks with accountable policy.
• Overly broad allowlists: Can admit the wrong dataset; overly narrow allowlists may block legitimate evolution.
• Approval by filename: Renaming an object to approved.csv provides no trustworthy evidence.
• Using `latest` after approval: Causes time-of-check/time-of-use races.
• Rechecking only pipeline status: A completed ingestion job does not imply its dataset was approved for this training purpose.
• Ignoring revocation: A snapshot approved last week may have since lost permission for new training use.
• Trusting checksum alone: Matching bytes does not establish data quality or usage authorization.
• Treating model-registration approval as data approval: These are different gates at different stages.

In short: Admit training data only through an immutable snapshot receipt with verified freshness, quality, approval, lineage and access scope. Enforce the decision in the orchestrator *and* the data loader, block unknown or revoked versions, and audit the actual bytes and identity consumed by every model-training run.