How can dataset snapshots be made reproducible across training runs?

A reproducible dataset snapshot is an immutable, uniquely identifiable copy or logical version of the exact data used by a training run. When the run is repeated, it must resolve to the same records, values, and schema—even if the source database has changed.

Questions 72 and 73 introduced DVC's Git linkage and shared storage. Here the emphasis is snapshot design independent of the versioning tool: immutability, manifest files, checksums, data splits, and exact run references.

Beginner-friendly example

A food-delivery team trains a delivery-time model every week.

• On Monday, the approved dataset contains 100,000 orders. The team publishes snapshot `orders-2026-w44-v1`.
• On Wednesday, 5,000 more orders arrive in the source database.
• A developer reruns Monday's experiment on Friday.

Incorrect approach: Query SELECT * FROM orders again. The input now contains 105,000 rows, so this is not the original experiment.

Correct approach: Read immutable snapshot orders-2026-w44-v1 with exactly 100,000 approved rows. Its version identifier and integrity metadata match the original run. The extra 5,000 rows belong in a new snapshot, not the old one.

Reusing the same dataset does not guarantee identical model weights: preprocessing code, random seeds, training environment, accelerator behavior, and configuration also affect reproducibility.

Six practical requirements

1. Freeze the input at a well-defined boundary. Use a committed table version, a transactionally consistent database export, or files written to an isolated snapshot directory. Do not read multiple changing tables at unrelated times and assume they form one coherent snapshot.

2. Assign an immutable version identifier. Use a unique snapshot ID or storage version, such as orders-2026-w44-v1, not a mutable path named latest. A timestamp alone is useful context but not proof of content identity.

3. Publish a manifest. Record the dataset ID, creation time, data range and cutoff, schema version, row count, file list, file sizes, and relevant content digests. For a multi-file dataset, record checksums per file and optionally a deterministic aggregate manifest hash.

4. Verify integrity on read. Compute and compare a cryptographic checksum such as SHA-256 over the actual saved bytes. A matching digest strongly indicates the stored bytes have not changed. If multiple files are used, check all listed files and their identities. A checksum is an integrity check, not a substitute for quality validation.

5. Preserve training/validation/test membership. Record split membership explicitly or use a deterministic split method with fixed input ordering, seed, group/time boundaries and code version. A seed alone may not preserve splits if the row order or dataset changes.

6. Capture the effective run inputs. Save snapshot ID, source cutoff, preprocessing version, feature definitions, parameters, training/evaluation split identities, container or environment version, and model artifact reference with every experiment run.

Simple snapshot manifest

JSON
{
  "dataset_id": "delivery-orders",
  "snapshot_id": "orders-2026-w44-v1",
  "row_count": 100000,
  "schema_version": "orders-schema-v3",
  "data_cutoff_utc": "2026-11-01T00:00:00Z",
  "files": [
    {
      "path": "part-000.csv",
      "sha256": "<real SHA-256 digest generated at publish time>"
    }
  ],
  "split_version": "time-split-v2"
}

The digest above is a placeholder. In production it must be populated from the saved file, never invented.

Calculate and verify a checksum in Python

PYTHON
import hashlib

def sha256_file(path):
    digest = hashlib.sha256()
    with open(path, "rb") as f:
        for chunk in iter(lambda: f.read(1024 * 1024), b""):
            digest.update(chunk)
    return digest.hexdigest()

expected_digest = manifest["files"][0]["sha256"]
actual_digest = sha256_file("part-000.csv")
if actual_digest != expected_digest:
    raise ValueError("Dataset snapshot integrity check failed")

Here manifest must be loaded from a trusted, previously published manifest containing the real digest. This code checks saved bytes, not whether the schema, labels, or records are meaningful.

Numeric illustration: why a snapshot matters

Suppose Monday's snapshot has 100,000 rows. An 80/10/10 split yields:

• 80,000 training rows
• 10,000 validation rows
• 10,000 test rows

If Friday's query unexpectedly includes 105,000 records, an 80/10/10 split yields 84,000 / 10,500 / 10,500, and potentially different individual rows in each group.

Even if you keep the same random seed, these are not the same training and evaluation datasets. Explicit snapshot and split identities remove the ambiguity.

For time-series prediction, the snapshot and split policy should also prevent future labels or other information from leaking into earlier training periods.

Where can snapshots live?

• Versioned object storage: Write immutable files under a versioned prefix; retain each referenced object version and use a manifest to capture the exact set of files.
• Versioned lakehouse tables: Delta Lake and similar table formats offer table versions or snapshots that let readers query a historical state. Retention and vacuum/cleanup policies must preserve versions required for reproducibility.
• Transactional database exports: Export records at a consistent cutoff or database snapshot, then retain the materialized result and metadata.
• Dataset versioning platforms: Tools such as DVC can retain identifiable datasets, but only if the relevant content remains available.

An object-store *folder name* does not automatically make the files immutable. Enforce access controls, write-once or retention policies, and safe publication rules where required.

How do you publish safely?

1. Create the output in an isolated staging location.
2. Finish writing all files.
3. Validate schema, counts, null rates, and business rules.
4. Compute real file checksums and prepare a manifest.
5. Publish the snapshot reference or manifest as the final committed step.
6. Prevent subsequent mutation and enforce retention for the required period.
7. Record the snapshot ID in each training and evaluation run.

Readers should only consume a committed snapshot, never a partially written directory.

What tools can record the identity?

MLflow supports logging dataset source, digest, schema and profile alongside a run. Be careful: not every MLflow dataset digest is a cryptographic hash of all raw file bytes; some dataset types use metadata or sampling. For a strict byte-for-byte guarantee, separately maintain actual file checksums or an authoritative storage version.

Delta Lake's table time travel can select a historical version, provided the old transaction log and underlying files are still retained.

Common mistakes

• Using latest as the only dataset reference.
• Keeping a snapshot ID but allowing its files to be overwritten.
• Saving a CSV while failing to preserve its source cutoff, schema, or splits.
• Assuming that checking the row count proves that every value is identical.
• Deleting old table versions or files before training-run retention expires.
• Recording a dataset digest without knowing what exactly the digest covers.

In short: Make datasets reproducible by committing immutable snapshots with unique IDs, real integrity hashes, consistent data cutoffs, explicit splits, and retained storage. Every training run should record exactly which snapshot it used. The next question connects these snapshots to the full source-to-feature-to-model lineage.