How do you capture provenance for models trained using external datasets?

Provenance is a verifiable record of where training data came from, who supplied it, which exact version was used, how it was transformed, and which model was produced from it. For external datasets, it should also record the license and contractual permissions, because being able to download data does not necessarily grant permission to train, redistribute, or serve a commercial model with it.

This lesson focuses on traceable origins and model-to-source relationships. Question 111 covered whether a dataset is approved to enter training, and Question 112 covered coordinating multiple update schedules. Provenance answers a different question: Can we explain and audit the exact external inputs that influenced this model months or years later?

Beginner-friendly example: external weather data for delivery ETAs

A delivery company predicts arrival times using its own order records plus an external historical weather dataset.

The pipeline receives:

• Internal orders: 80,000 labeled deliveries from company-owned snapshot orders-v42.
• External weather: 20,000 hourly weather observations from a third-party provider.
• Training result: ETA model eta-v18 produced by training run run-20261106-018.

The company should save:

• Provider identity: Legal provider/operator name, verified service, dataset product ID and accountable internal owner.
• Permission evidence: Exact license/contract revision, allowed commercial training purpose, geographic scope, expiry, redistribution limits and attribution obligations.
• Source identity: Provider URL or dataset repository, exact release or commit ID, retrieval timestamp and object versions.
• Integrity: SHA-256 digest of the exact archived source bytes and a manifest of every file.
• Transform lineage: Versioned join, filtering and aggregation steps that combined weather with order data.
• Quality evidence: Validation run ID, outcome, row counts, expected coverage and relevant limitations.
• Training linkage: Immutable processed dataset ID, split IDs, training run, model artifact digest and registered model version.

A mutable URL such as https://provider.example/weather/latest.csv is not sufficient provenance: tomorrow it may deliver different contents under the same name.

Numerical example: reconcile source-to-training lineage

Suppose the weather supplier provides 20,000 rows in an archived snapshot:

• 20,000 rows downloaded and their byte-level checksum recorded.
• 800 duplicate source rows removed under an agreed deduplication policy.
• 200 rows rejected because of missing mandatory event timestamps.
• 19,000 rows remain in the cleaned weather dataset.

Arithmetic: 20,000 - 800 - 200 = 19,000.

The source-to-cleaning retention is 19,000 / 20,000 × 100 = 95%; 5% was removed by these transformations. The denominator is the *weather* dataset, not the 80,000 delivery orders. These counts are illustrative and assume the 800 duplicates and 200 invalid-time rows do not overlap. If checks overlap, reconcile mutually exclusive reason counts.

After a point-in-time join, suppose 76,000 of 80,000 orders obtain a valid weather feature:

• Join coverage = 76,000 / 80,000 = 95%.
• Unmatched orders = 4,000, or 5%.
• The pipeline should preserve the join policy for those 4,000 orders: rejected, filled with a defined missingness indicator or imputed under an approved procedure.

The final processed training snapshot gets its own checksum because cleaning and joining create new content. The original source checksum cannot prove that the final training table was built correctly.

The minimum provenance chain

External provider / licensing authority
                  |
       Source release + immutable bytes
                  |
      Download / verification activity
                  |
     Raw archived external snapshot
                  |
        Cleaning / transformations
                  |
    Processed weather dataset version
                  |
       Point-in-time join with orders
                  |
        Final training dataset version
                  |
           Model training run
                  |
       Model artifact + registry version

In W3C PROV terminology, entities include dataset snapshots and models, activities include downloading and transforming data, and agents include the source provider and pipeline operator. Record what each activity *used* and what it *generated*. OpenLineage supports dataset versions, input/output datasets, job/run information and facets for quality measurements. <See the linked official references.>

Record the exact external source, not merely its homepage

For each external dataset capture:

• Provider legal name, dataset title, product ID and versioned citation.
• Distribution URL, original filename or object-store key and the retrieval method.
• Commit hash, immutable release, object version or vendor snapshot identifier.
• UTC retrieval timestamp and, where relevant, the provider's data-coverage interval.
• Source document/version, API request parameters and source pagination identifiers, if reconstruction requires them.
• Total bytes, file count and per-object checksums.
• Original schema and a documented collection description or dataset card.

For datasets hosted on Hugging Face, use a pinned commit revision rather than main when you require an identical source revision. A commit reference identifies a repository state, while an independently verified byte digest protects the exact downloaded content. The provider can still later remove or restrict access, so choose an archive policy consistent with the license.

Licensing and permissions are part of provenance

Capture evidence, not just an unverified label such as license = open:

• Exact SPDX expression when applicable, plus a pointer or archived copy of the governing license text/version.
• Contract or data-processing agreement ID for commercial feeds.
• Whether training, derivative artifacts, redistribution, model deployment and cross-border processing are permitted.
• Attribution, citation, audit, usage reporting or deletion requirements.
• Personal-data restrictions and consent/legal-basis requirements where relevant.
• License expiration, upstream revocation mechanism and an internal accountable approver.

Caution: A license covering the dataset does not automatically settle all rights in personal data, third-party contributions, or model outputs. Seek appropriate legal review for ambiguous cases. Avoid publishing provider credentials, private contracts or protected source rows in public experiment metadata; store restricted evidence behind access controls and reference it by an internal document ID.

Example versioned provenance manifest

YAML
manifest_version: "1"
training_run_id: "run-20261106-018"
model_version: "eta-v18"

external_source:
  provider: "Example Weather Provider"
  dataset: "Historical Hourly Weather"
  release_id: "weather-2026-10-r3"
  source_revision: "commit-or-vendor-revision-id"
  retrieved_at_utc: "2026-11-06T05:30:00Z"
  raw_snapshot_id: "weather-raw-v7"
  sha256: "<verified-full-64-character-sha256>"
  row_count: 20000

rights:
  permission_record_id: "rights-2026-184"
  license_version: "agreement-v2"
  permitted_purpose: "internal-eta-model-training"
  reviewed_by: "data-governance-team"

lineage:
  clean_transform_commit: "git-abc123"
  processed_snapshot_id: "weather-clean-v9"
  cleaned_row_count: 19000
  orders_snapshot_id: "orders-v42"
  join_pipeline_commit: "git-def456"
  final_training_snapshot_id: "eta-training-v51"
  join_matched_orders: 76000

validation:
  quality_run_id: "weather-quality-889"
  status: "PASS"

model:
  training_code_commit: "git-987xyz"
  artifact_digest: "<verified-model-digest>"

This is a generic design example, not a formal W3C/OpenLineage YAML specification. The placeholders are intentionally not valid integrity hashes. Real manifests must be bound to trusted permissions, snapshots and metadata records.

Capture file checksums correctly

A simple integrity helper for files received from a provider:

PYTHON
from pathlib import Path
import hashlib

def sha256_file(path: Path) -> str:
    digest = hashlib.sha256()
    with path.open("rb") as file:
        for chunk in iter(lambda: file.read(1024 * 1024), b""):
            digest.update(chunk)
    return digest.hexdigest()

source_file = Path("staging/weather-2026-10.parquet")
source_digest = sha256_file(source_file)
print(source_digest)

Save the digest with the original archived file identity, its size, provider release and retrieval receipt. Verify it again before training or reprocessing. For a dataset with multiple files, record individual hashes and an ordered/canonical manifest digest; simply concatenating filenames from an unordered listing can produce unstable results.

A matching digest proves byte-level identity given a trusted expected digest. It does not prove provider authenticity, legal permission, data quality, or that transformations were correct. Verify source authenticity independently and protect provenance metadata against unauthorized edits.

Connect provenance to MLflow training runs

MLflow supports tracking datasets with name, source, digest, schema and profile, and linking them to training runs through mlflow.log_input(). <See MLflow dataset-tracking documentation.>

For a DataFrame that has already been loaded from a pinned, authorized source:

PYTHON
import mlflow
import mlflow.data

# df_weather must be loaded earlier from a verified immutable snapshot.
dataset = mlflow.data.from_pandas(
    df_weather,
    source="s3://example-archive/weather/release-2026-10-r3/",
    name="external-weather-2026-10-r3",
)

with mlflow.start_run(run_name="eta-train") as run:
    mlflow.log_input(dataset, context="training")
    mlflow.set_tags({
        "external_provider": "Example Weather Provider",
        "source_release": "weather-2026-10-r3",
        "rights_record_id": "rights-2026-184",
        "quality_run_id": "weather-quality-889",
        "final_training_snapshot": "eta-training-v51",
    })
    # Fit the model and log its artifact via a supported MLflow model API.

The DataFrame df_weather is illustrative and must already exist. This snippet logs metadata; it does not download, checksum-verify, approve, or archive the source. Also, an MLflow dataset digest is an implementation-defined dataset fingerprint and should not be assumed equivalent to a cryptographic SHA-256 over archived source bytes. Store an independent source-file checksum.

Log the final joined training dataset as a separate input when applicable, plus its source lineage and split IDs. The external weather input alone does not describe the complete training population.

OpenLineage for transformation provenance

A pipeline can emit OpenLineage job/run events describing:

• Ingestion job input: provider snapshot.
• Ingestion job output: raw archived dataset.
• Cleaning job input/output: raw to cleaned dataset.
• Feature-building job inputs: cleaned weather and internal orders.
• Feature-building job output: immutable joined training snapshot.
• Training run input: training snapshot; output: model artifact/metadata.

Use standard dataset version and dataQualityMetrics/dataQualityAssertions facets where supported. Include job/source-code versions and ownership. Lineage events allow an impact query such as: Which model versions depend on external weather release r3? Tools differ in which facets they capture automatically, so verify actual end-to-end coverage and fill gaps explicitly.

Auditing, change tracking and source revocation

Suppose an external provider announces that release weather-2026-10-r3 contains corrupted temperature readings or the license is withdrawn.

A useful provenance system can:

1. Locate the provider and affected exact release ID.
2. Find raw/cleaned datasets derived from it.
3. Traverse transformation links to all dependent training snapshots.
4. Identify training runs and registered/deployed models that consumed those snapshots.
5. Determine whether the issue affects use permissions, integrity, accuracy, or all three.
6. Open an incident and pause new use of the affected dataset versions.
7. Follow contractual/legal direction for retention, deletion or retraining and document the decision.
8. Rebuild from an authorized corrected release when appropriate, then re-evaluate downstream models.

For example, if release r3 contributed to 4 training snapshots, each used by 3 experiment runs, the maximum direct run impact is 4 × 3 = 12 runs under the assumption of distinct run relationships; actual affected models must be deduplicated because one model may have multiple linked experiments. Track exact reverse edges instead of guessing that every current model used the latest external data.

Revocation does not always mean every previously trained model must be deleted automatically; duties vary with licensing, privacy, contract and technical circumstances. Make that determination through the appropriate review process rather than guessing from a dataset tag.

Reproducibility and retention trade-offs

• Only keep external URL: Cheap, but weak; the link can change, disappear or return a new revision.
• Archive complete permitted source bytes: Best for later byte-for-byte checks, but requires storage, access controls and permission to retain copies.
• Store hash + manifest without bytes: Offers identity and traceability, but cannot recreate the dataset if the provider removes it.
• Track transformations and split IDs: Improves model reproducibility but adds metadata to maintain.
• Record every column-level dependency: Valuable for impact analysis, but can be expensive; prioritize material features and sensitive sources.
• Use cryptographic signatures/attestations: Stronger tamper evidence when implemented correctly, but not a substitute for trusted provider identity or rights review.

Coordinate provenance retention with artifact retention: if a production model remains in service, preserve the legally permitted provenance evidence needed to explain and govern it. Do not retain external data longer than applicable permissions allow simply because it would be convenient for reproducibility.

Common mistakes

• Logging weather/latest.csv without a source release or immutable snapshot.
• Saving only an MLflow dataset digest and assuming it proves original file integrity.
• Tracking data provenance but omitting licensing/usage permissions.
• Preserving only the raw source while losing the transformation code and split definition.
• Treating a correct hash as evidence that the dataset is ethically or legally usable.
• Assuming a public download URL automatically grants commercial training rights.
• Logging protected license documents or confidential data into broadly visible experiment metadata.
• Copying current data into a historical run while labeling it with an older release ID.
• Failing to link derived datasets to model versions, making incident impact analysis impossible.
• Automatically discarding original run lineage when an upstream dataset is revoked.

In short: A good external-data provenance record binds provider, rights, exact source revision, original bytes, transformation history, quality evidence, final training snapshot and model run. Capture it at ingestion and every important transformation boundary so audits, reproducibility and incident investigations can follow the full chain.