MLOps
MLOps applies software engineering and operations discipline to machine learning. It covers reproducible training, model registries, deployment, monitoring, governance, and incident response.
How do you build automated dataset quality checks into ingestion jobs?
Automated dataset quality checks are repeatable tests that run inside data ingestion jobs before a dataset is published for ML training. They measure whether an entire batch meets agreed completeness, uniqueness, validity, referential integrity, and statistical expectations. A failed critical check should stop publication, even if the ingestion program itself completed without error. Question 100 explained how to quarantine individual invalid records . This question focuses on dataset level validation rules, quality scores, release gates, monitoring, and automated orchestration . Beginner friendly example: daily delivery orders A food delivery company ingests a daily batch of 10,000 orders . Its ML model predicts delivery time. Before a training dataset becomes available, the ingestion pipeline evaluates: Completeness: At least 99% of rows contain restaurant id. Uniqueness: Every non null order id must be unique. Validity: At least 99.5% of distance km values must be finite and within 0–100 km . Referential integrity: At least 99.8% of non null restaurant identifiers must exist in the current, appropriately versioned restaurant reference dataset. Statistical checks: Daily row count must be within an approved range of 8,000–12,000 . A large shift in distance distribution should trigger further investigation. The batch reports: restaurant id present in 9,940 / 10,000 = 99.4% → PASS . order id has 5 duplicate IDs → FAIL the zero duplicate rule. Valid distance km in 9,970 / 10,000 = 99.7% → PASS . Matching restaurant IDs in 9,925 of 9,940 non null IDs ≈ 99.85% → PASS . Row count 10,000 → PASS . Result: block promotion of this batch as a fully validated training dataset. Four passing tests do not cancel a critical uniqueness failure. The owner should investigate the duplicated IDs and rerun the relevant checks after a controlled correction. The denominators and policy exceptions need to be defined in advance; a completeness rate and a referential integrity rate can legitimately use different eligible populations. What quality dimensions should a job check? 1. Completeness Measure missing or unusable values for required columns. For example: completeness = non null required values / total rows Set thresholds by column criticality. A training label may require 100% completeness in a labeled dataset, while an optional explanatory feature may tolerate some missing values if the model handles them. 2. Uniqueness Check keys that should identify exactly one entity or event. Count duplicate keys, accounting for whether repeated corrections or multiple valid events are part of the source semantics. A uniqueness percentage is useful for monitoring, but a critical event ID may need a strict zero duplicate rule. 3. Validity Test allowed ranges, units, accepted categories, timestamp formatting, finite numeric values, and domain specific relationships such as delivery completion not preceding pickup. A structurally valid float may still be semantically invalid: distance km = 3 is numeric but not a reasonable physical distance for this example. 4. Referential integrity Check that foreign keys reference valid entities, for example orders.restaurant id must appear in the appropriate restaurant reference snapshot. Match against the right snapshot or version . Comparing yesterday's orders with an arbitrarily updated restaurant table can create false failures when restaurants were deleted or their IDs changed. 5. Statistical health Monitor row count, null rate, mean/median, standard deviation, category frequencies and, where appropriate, distribution shifts relative to a trusted baseline. For example, historical daily order counts around 10,000 may make a batch of 2,000 suspicious. Statistical thresholds require context: holidays, growth and seasonality may legitimately change volume. A distribution anomaly should sometimes warn rather than automatically block, depending on the risk and validation policy. Embed the checks directly in the ingestion job Publication is the gate: Do not make downstream training discover a half written or unvalidated dataset. Write candidate output to a staging location, evaluate it, then atomically publish an approved manifest or update a trusted dataset pointer when storage supports it. An AWS Glue Data Quality implementation AWS Glue Data Quality lets ETL pipelines express rules in Data Quality Definition Language (DQDL) . Its documented rule types include completeness, uniqueness, ranges, and referential integrity. An illustrative ruleset is: This checks count, completeness, uniqueness and range; it is not the complete five dimension policy. For thresholds on the proportion of valid distance values, use the appropriate rule semantics and supported threshold configuration instead of treating an all row range rule as a 99.5% threshold. Referential integrity checks additionally need a correctly configured reference dataset and can be supported through the Glue Studio data quality transform. Configure the Glue ETL job to publish quality results and metrics, then enforce your explicit failure policy . Do not assume that generating a low quality score automatically fails the entire ingestion job ; configure the desired stop/publish behavior. Great Expectations integration In modern Great Expectations (GX Core), the workflow is: 1. Define an Expectation Suite containing the dataset's required rules. 2. Bind it to an ingestion Batch Definition through a Validation Definition . 3. Run the validation on the staged batch. 4. Inspect the result and explicitly prevent publication if critical expectations fail. 5. Optionally use a Checkpoint and Actions to save results, update Data Docs or notify owners. Once the Data Context and Validation Definition are configured: The batch parameter keys depend on the configured Batch Definition; publish approved dataset is application pseudocode. In production, distinguish critical failures from warning only checks rather than assuming every warning should stop ingestion. Persist a reference to the actual validation result, dataset snapshot and rule suite version. Configurable thresholds and policy Separate quality rules into severity levels: Critical: Missing mandatory labels, duplicate primary keys, incompatible source schema or failed referential integrity above the approved tolerance → block publication. Warning: Small shifts in average feature values, a modest drop in optional field completeness, or a borderline but explainable volume change → publish only if the operational policy permits and alert. Informational: Category distributions, average distances and other baseline statistics → observe trends and refine quality policy. Do not loosen thresholds automatically just to make a failing job green. Version threshold changes and require appropriate review. Monitor quality across daily jobs Track: Per rule pass/fail , observed statistic, threshold and severity. Batch identity: Input snapshot, output version, processing run ID and validation suite revision. Trend: Completeness, duplicates, reference match rate, row count and feature statistics over time. Pipeline outcome: Blocked versus published batches, delays to approved training data, failures and retries. Owners and alerts: Send actionable failure reasons to the responsible data or ML team. Example: if completeness drops from 99.8% to 99.2% and then to 98.7% , the final job breaches a 99% threshold , but the earlier downward trend could have supported a warning before the failure. AWS Glue can publish data quality results and CloudWatch metrics, and GX can persist validation results and run Checkpoint Actions, depending on the implementation. Failure handling and recovery 1. Validation rule fails: Mark staged data as unapproved, record the offending metric and alert the owner. Do not silently continue training. 2. Validation service fails: Treat missing evidence as unknown/failure , not as a passed quality check; retry transient infrastructure errors. 3. A reference dataset is unavailable: Do not falsely report that referential integrity passed. Retry or stop according to dependency policy. 4. Quality rules need revision: Review and version the change; validate it against representative old and new batches. 5. After data correction: Re run the complete required checks on a new immutable candidate dataset; publish only a passing version. 6. Historical reprocessing: Keep rule and dataset versions, because replaying an old batch under today's rules may produce a different outcome. The mechanics for isolating and replaying individual invalid rows belong to Question 100; a dataset quality gate instead answers whether a whole candidate dataset can be trusted for its intended consumer . Common mistakes Checking only that the ingestion job executed successfully. Treating five passed rules out of six as automatic approval when the failed rule is critical. Hardcoding a data quality percentage without a clear eligible row denominator. Comparing referential keys with the wrong reference data snapshot. Using generic statistical alarms that fire every holiday without a controlled exception process. Running checks after the dataset has already been published to training consumers. Reporting a quality score but not enforcing the appropriate pipeline gate. Failing to save rule versions and test evidence for audit and reproducibility. In short: Validate staged datasets automatically using completeness, uniqueness, validity, referential integrity and statistical rules . Set reviewed thresholds, distinguish critical failures from warnings, block publication of untrusted data, and monitor results across runs. The next lesson covers responding when upstream data is missing or delayed.
How do you detect and respond to missing or delayed upstream data?
Missing or delayed upstream data is an operational dependency failure. An ML pipeline should know exactly which dataset or event window it expects, when it should be available, how completion is proven, and what to do if that expectation fails. Do not start production retraining on an incomplete or unverified input. Unlike Question 101, which checks the quality of data already received , this lesson examines absent, late, partially published, or stalled upstream deliveries . Beginner friendly example: a late delivery orders snapshot The Orders Platform promises each previous day snapshot by 06:00 UTC . The ETA model's daily retraining starts at 06:30 UTC . For November 2, the training job expects: A committed partition orders/date=2026 11 02. A producer manifest listing 8 files and their integrity checks. Approximately 100,000 orders , although the exact expected count must come from the source or an agreed quality rule. A verified completion marker tied to the correct date and manifest revision. At 06:05 UTC on November 3 , only 6 of 8 files and 75,000 reported orders have arrived. File completion: 6 / 8 × 100 = 75%. Estimated record completion: 75,000 / 100,000 × 100 = 75%. SLA delay so far: 5 minutes . Decision: Do not mark the partition ready; hold dependent training and notify the upstream owner. The two 75% figures happen to match in this example. File count is not a reliable proxy for record completeness unless every file's expected contents are known. Suppose the producer completes and commits the snapshot at 06:22 UTC . The dataset was 22 minutes late , but it may still pass validation before the scheduled 06:30 training start. Record the upstream SLA breach even if downstream training ultimately starts on time. Four conditions worth distinguishing 1. Missing: A required partition, table snapshot or source event has not appeared. 2. Partially delivered: The destination exists, but the manifest, expected files, checksums or final transaction/commit record are incomplete. 3. Late: The dataset becomes ready after the contractual or internal arrival deadline. 4. Stalled: A source responds to queries, but its last committed partition or event time progress has stopped advancing. An intentionally empty dataset is not necessarily missing: the producer must explicitly publish a validated empty but complete partition rather than leaving consumers to guess. What should be monitored? Expected versus committed partitions: Logical partition ID, event date, source and dataset revision. Arrival latency: Verified completion time minus due time, with negative values representing early delivery. Completeness: Expected files/checksums, producer row count or transactional manifest, and related required inputs. Last successful update: Producer commit time and consumer ingestion completion. Source progress: High water mark or streaming event time watermark, with their distinct meanings. Blocked work: Waiting training jobs, delayed labels, impacted feature builds, and recovery time. Retry and alert history: Attempt count, alert recipient, incident ID and next escalation. Monitor source arrival separately from the dataset content quality checks covered in Question 101. A successfully committed partition may still fail its quality rules. Batch high water marks versus streaming watermarks For a batch source, a producer might publish committed through=2026 11 02 to indicate that all required daily partitions up to that logical date were finalized. Only trust this marker if the producer's protocol and authorization ensure it reflects a complete commit. In a streaming pipeline, a watermark estimates how far the stream has progressed in event time . Apache Beam uses watermarks and event time triggers to decide when windows are eligible to emit results; late events may still arrive after a watermark crosses a window end. For example, a five minute window ends at 10:05 . A watermark at 10:04 indicates the system has not yet passed the window boundary. When it advances to 10:06 , the window may be emitted according to its trigger configuration, but a record with an old event timestamp could still arrive later. A watermark is not absolute proof that every event has arrived. Allowed lateness, correction/retraction policies, producer completeness and downstream readiness must be designed explicitly. How to gate pipeline execution Use a ready/committed manifest rather than checking whether a directory merely exists. A process may create a folder long before it finishes writing files. A small Python readiness gate This illustrates the gate—not a secure, complete integration. Real code must verify manifest identity, expected partition, signature or access controls, checksum and commit metadata , and must reject stale commits. READY here means arrival checks passed , not that all dataset quality checks succeeded. Waiting and retries with Airflow Apache Airflow sensors can wait for objects, external tasks or custom upstream conditions. A sensor can have a finite timeout, sensible retry policy and an alert callback. mode="reschedule" on a compatible sensor releases its worker slot between checks instead of occupying it continuously. Supported deferrable sensors can suspend while waiting and later resume through the triggerer. For external sources, a custom sensor should read a trusted completed partition manifest, not rely only on existence of a prefix. Set a hard timeout and report the SLA breach separately: waiting until a sensor times out should not be the first indication that a 06:00 deadline was missed. An illustrative exponential retry schedule might check at 06:00 , 06:02 , 06:06 , and 06:14 , with jitter and bounded attempts in production. These checks may recover transient visibility or source delays; they cannot repair a producer that never published the partition. Which fallback is safe? Fail closed: Block retraining and keep the last validated model serving when the missing data is essential. This is generally safer than secretly training on partial labels. Use the previous approved snapshot: Only where a formally defined maximum staleness and downstream policy permit it. Record that a fallback was used and assess changes in data coverage. Proceed with partial data: Only after confirming minimum date/entity coverage, class balance, model risk and explicit approval. Missing days can disproportionately affect peak hours, rare cases or particular regions. Suppose seven daily partitions are required but only six are present. 6/7 × 100 ≈ 85.7% is the fraction of days , not necessarily the fraction of training examples or customers. Do not infer that a model trained on six days is representative. Backfill safely once data arrives 1. Confirm the recovered source partition is committed and passes its normal quality gates. 2. Pin the exact logical date, source snapshot ID and processing code revision. 3. Launch a targeted backfill for missing dates instead of using today's latest dataset. 4. Use a durable idempotency key such as eta train:2026 11 02:snapshot 123 to avoid duplicate training artifacts and publications. 5. Recompute dependent features or dates when the dependency graph requires it. 6. Independently validate the result and apply normal model approval checks before any deployment. 7. Record the incident cause, source owner, downtime, replays and downstream recovery verification. If the source later corrects records, decide whether dependent historical feature datasets and models must be rebuilt. Do not merge corrected and original snapshots without clear provenance. Incident response checklist Detect: Source due time missed, missing committed partition, stalled source watermark or incomplete manifest. Contain: Hold affected training runs and ensure serving continues with a trusted model. Diagnose: Check source producer status, storage writes, scheduler history, partition mapping, timezone handling and access failures. Coordinate: Alert the responsible upstream owner, track one incident for related downstream symptoms, and specify impacted logical dates. Recover: Retry transient failures, await a verified commit, perform controlled backfill and validate outputs. Prevent recurrence: Improve source completion signaling, early warnings, upstream SLAs, reconciliation and runbooks. Avoid routing hundreds of downstream task alerts to engineers when the root issue is one upstream partition: correlate them with the same source incident. Common mistakes Assuming a folder exists means all records are ready. Confusing an empty but committed dataset with a missing one. Ignoring timezone shifts that cause the wrong partition date to be requested. Treating a watermark as a perfect event completeness certificate. Silently using stale or incomplete training data. Retrying forever without alerting the owner. Running a backfill using mutable latest data rather than pinned historical inputs. Allowing repeated backfills to overwrite approved datasets or trigger duplicate model releases. In short: Define arrival expectations, verify committed partitions and progress, monitor SLA breaches, block critical dependent runs, and recover through auditable, idempotent backfills. Missing data is a reliability incident—not merely a failed validation check.
How do you decide which training pipeline tests should block execution?
A training pipeline test should block execution when a failure makes the next step unsafe, invalid, irreproducible, or very likely to waste expensive resources. Less critical failures can be warnings, but they must remain observable and have an accountable owner. The decision should be based on risk and where the test sits in the workflow , not simply on whether the test returned red. Question 102 covered missing or delayed upstream inputs. This lesson focuses on choosing the consequence of a failed test after an input or pipeline step is assessed. Three possible outcomes, not only pass/fail BLOCK TRAINING: Do not start or continue model fitting. Use for missing required labels, leaked target features, corrupt snapshots, incompatible tensors, missing authorization, invalid training code or other faults that make training untrustworthy. BLOCK PUBLICATION OR DEPLOYMENT: Training may finish for diagnosis or experimentation, but the resulting model must not be registered as approved, promoted, or served until evaluation and serving checks pass. WARN AND MONITOR: Proceed if the impact is bounded, policy explicitly permits it, and the team records the exception. Warnings must not silently become permanent acceptance of dangerous defects. A fourth useful state is UNKNOWN : a test could not run or its evidence is missing. For a mandatory safety or integrity gate, treat UNKNOWN as blocked , not as PASS. Beginner friendly example: a food delivery ETA training job A company trains a model on 100,000 delivery orders . The daily job runs on a GPU and takes about 90 minutes . The team defines illustrative gates: Required label coverage: At least 99% of training examples have valid delivery duration min labels. Critical; block training if unmet. No target leakage: Protected leakage tests must pass. Critical; block training. Training/serving feature schema: Exact expected feature names, types and units. Critical; block training when incompatible. Resource readiness: Enough quota/storage to safely finish the run. Block or defer before expensive fitting. Model validation MAE: At most 4.0 minutes on a fixed holdout. This blocks model promotion , but the model may still finish training so the team can inspect results. Minor optional field completeness: At least 95% , with an approved fallback for missing values. A small breach could be a warning if impact was evaluated and policy allows it. Assume 98,400 of 100,000 labels are valid: 98,400 / 100,000 × 100 = 98.4%. This is 0.6 percentage points below the critical 99% requirement, so the pipeline must BLOCK TRAINING , despite every other check passing. If instead label coverage is 99.5% , training can proceed when all other pre training gates pass. Suppose the trained candidate then scores 4.3 minute MAE against a required 4.0 minute maximum . Its training run may be retained, but promotion is blocked . The distinction avoids wasting investigation evidence while protecting production. All thresholds above are hypothetical. Adopt limits appropriate to the model, business impact, data distribution and available capacity. How to choose which tests block 1. Assess the failure's consequence Could it produce incorrectly labeled examples, leak future information, expose protected data or break an explicit contract? Could it create an unsafe or unusable artifact? Could it waste substantial compute without producing any useful result? Could it undermine meaningful evaluation or production behavior? 2. Assess likelihood and detectability How frequently does this failure occur? Would it be caught by a later independent gate? Would proceeding hide the defect or make it difficult to reproduce? 3. Choose the earliest effective enforcement point Fail fast before training for invalid inputs or missing resource prerequisites. Stop during training for non finite losses, fatal runtime errors or exhausted resources. Fail after training, before publication/promotion for unacceptable holdout metrics or incompatible serving artifacts. Fail before deployment for missing approval, critical security issues or serving smoke test errors. 4. Assign severity, response and owner Each policy should define gate id, severity, stage, threshold, evidence source, owner, exception policy and failure action. A failed critical gate must not become a warning just because retraining is overdue. A simple risk heuristic is severity of harm × probability × difficulty of detecting the error later . These dimensions can guide prioritization but are not a universal numerical standard; privacy/security or correctness violations may be mandatory hard blocks irrespective of a calculated score. Typical tests and the right control point Data integrity and leakage Missing essential labels, unresolved source integrity issues, wrong join keys or proven target leakage: block training . A small variation in an approved optional field's null rate: potentially warning , if the model handles missingness. A missing critical dataset or unverified snapshot: block/defer training , consistent with Question 102's readiness gates. Training correctness Incorrect feature order, tensor shape, bad class mapping, missing dependency version, numerical overflow/NaN loss: block/fail execution . Training loss noisier than usual but otherwise within a documented tolerance: investigate, and possibly warn . Resource and budget controls Insufficient GPU memory, storage quota or required credentials: block/defer before spending resources. Forecast cost above a configured hard spending cap: block or require explicit budget approval . Resource usage trending higher but still within an approved budget: warn , monitor and consider optimizing. Model evaluation Candidate underperforms required global or critical slice thresholds: block promotion , even if the training run succeeds. Small noncritical metric regression inside an approved tolerance: warn , with owner review if needed. Serving and release Unloadable model, broken input signature, missing required approval or major vulnerability: block deployment . A cosmetic documentation issue may be a warning for a low risk internal experiment, but could be a hard requirement for a regulated release. The same test can legitimately have different enforcement behavior at an exploratory notebook run versus a scheduled production training or release job. Keep the policies explicit. Simple policy as code example This is an educational gate, not a secure release service . Evidence must be bound to a specific immutable dataset/run and come from trusted validation systems. Real code should reject booleans as numeric values, validate ranges 0..1, and persist the evaluation outcome. A pipeline must enforce can train through the orchestrator, not merely print a report. Model quality gate after training MLflow supports evaluating candidate models and validating metrics against predefined thresholds with mlflow.validate evaluation results(). For example, where MAE is the metric: The precise metric name must match the evaluation result's keys, and candidate result must exist. This gate protects promotion , not necessarily the execution of model fitting. MLflow's documentation notes the modern dedicated validation API introduced around MLflow 2.18.0. Warning policies must be deliberate A warning requires: A documented threshold, why exceeding it is tolerable and which environments allow it. An owner and response deadline. A measurable trend or alert, not just a line in logs. Evidence that the model or pipeline can handle the degraded condition. A clear rule for converting repeated warnings into a blocking incident. For example, optional feature completeness falling from 96% to 94% could be a warning once. Remaining at 80% for a week should not be ignored merely because the first incident was classified as warning. Define escalation policies rather than relying on informal judgment. Exceptions and emergency overrides Rare exceptions may be needed, such as running a nonproduction diagnostic retraining job despite a known optional data issue. Require: 1. A named approver distinct from the requester when applicable. 2. The exact affected gate, dataset/run, risk explanation and scope. 3. An expiry time and explicit permitted action. 4. An audit record, alert and follow up task. 5. Protection against bypassing nonwaivable security, privacy or critical integrity gates. An exemption to run a diagnostic training job must not automatically authorize production deployment . Scope each override to the correct stage and artifact. Make failures actionable and auditable Persist every gate decision with: Pipeline and run IDs, code commit and immutable input snapshot. Gate version, thresholds, observed values, severity and outcome. Whether execution stopped, training completed or publication was blocked. Owner, exception approver, timestamp and rationale when relevant. Links to logs, evaluation reports and follow up incidents. Build dashboards for critical failure counts, failure causes, warning frequency, wasted compute prevented, false positive gates and recurring exemptions. Review whether gates are actually preventing incidents or merely creating noise. Google's ML guidance recommends independently testing training and serving infrastructure, input and feature correctness, integration and candidate model quality. Those categories provide a good checklist, while severity and enforcement policy must be adapted to the system. Question 104 will distinguish unit, integration and end to end test levels without repeating these release decisions. Common mistakes Treating every failed check as an automatic training halt, including post training evaluation metrics that should instead block promotion. Letting a critical leakage or label integrity failure become a warning to meet a deadline. Proceeding when mandatory test evidence is missing. Hardcoding thresholds without units, datasets, or sample size context. Treating aggregate metrics as sufficient when critical user segments regress. Printing an error without wiring the gate into the pipeline scheduler. Making overrides permanent or granting one approval the power to bypass all release stages. Testing only data correctness while ignoring model loadability, cost and environment readiness. In short: Block early for invalid inputs, block promotion for unacceptable model results, and warn only for bounded, owned risks. Keep decisions versioned, evidence based, enforced by the orchestrator and subject to tightly controlled exceptions.
How do you separate pipeline unit tests, integration tests, and end-to-end tests?
Separate ML pipeline tests by how much of the system they exercise: unit tests check one small piece of logic in isolation, integration tests check that connected components work together, and end to end (E2E) tests run a representative workflow from input to final output. All three are useful because they catch different classes of defects. Question 103 covered which failures block training or release . This lesson covers the scope, dependencies, execution speed and diagnostic value of each test layer . Question 105 will explain how to design small deterministic fixtures in detail. Beginner friendly example: delivery time training pipeline Suppose a food delivery company's ML pipeline contains six tasks: The model predicts delivery time using travel distance, historical restaurant delay and traffic conditions. Unit test: Verify that a feature transformation converts distance m = 2500 to distance km = 2.5. This test should not call the production warehouse, the model registry or the orchestrator. Integration test: Write a tiny batch into an isolated test database or object store sandbox, run the real ingest and feature building interfaces, and confirm that the downstream component retrieves distance km = 2.5 with the expected entity and column types. This catches connection configuration, serialization and schema mismatches that a mocked unit test cannot. End to end test: Execute the full DAG in a staging or local test environment against a small controlled dataset. Verify that it produces a loadable model artifact, evaluation report, tracked run metadata and candidate registry entry. Do not promote the test artifact to production. The same mathematical conversion appears in more than one layer, but each layer asks a different question : Does the function work? Do connected systems exchange the correct value? Can the whole workflow complete and produce trusted outputs? 1. Unit tests: isolate small logic Unit tests focus on one function, component or DAG structure rule. Scope: Individual transformations, validation helpers, date/partition calculations, feature name mapping and simple branching logic. Dependencies: Avoid network access, remote databases, real cloud credentials and expensive model training. Isolation: Replace external boundaries with fakes or mocks when the interaction itself matters. Speed: Usually milliseconds or seconds per test, enabling rapid feedback. Failure diagnosis: Precise; a failed assertion usually identifies the faulty function or contract. Best placement: Every developer change, pull request and pre merge CI build. An illustrative Python unit test: These tests do not need a DAG run or a data warehouse. A mock is helpful when an external API must be called; it should not replace the calculation being tested. For Airflow, unit level checks can also assert that a DAG loads and has the correct task IDs and dependency edges. This checks the graph definition , not whether the DAG successfully executes against its infrastructure. 2. Integration tests: exercise real boundaries Integration tests verify interactions between at least two components and often use a temporary real service. Scope: Data reader ↔ storage, feature builder ↔ database, trainer ↔ artifact store, model registry ↔ deployment interface or orchestrator task ↔ metadata database. Dependencies: Prefer ephemeral/local instances, test containers, temporary cloud resources or a properly isolated staging environment. Isolation: Stub only the dependencies outside the integration under test. If the goal is to verify a PostgreSQL query, use a real compatible database rather than mocking its query results. Speed: Typically slower than unit tests because setup, I/O, container startup and services add cost. Failure diagnosis: Narrower than a full pipeline but broader than one function. Best placement: Pull request CI for affordable important boundaries; additional suites before staging/release. Example integration scenario: 1. Store two valid delivery rows in an isolated test dataset. 2. Invoke the real ingestion/feature task. 3. Read the output through its real supported storage interface. 4. Assert schema, order/entity IDs, row count, units and committed output path. 5. Tear down the isolated resources. That test can uncover a database column called distance miles being mistakenly mapped to distance km even though the pure unit conversion helper passed. For Airflow, dag.test() can execute a real DAG run in a single serialized Python process, but requires a configured metadata database and a serializable DAG in a DAG bundle . It is therefore an integration style tool, not an isolated unit test. Its behavior also depends on the Airflow version and configured environment. 3. End to end tests: verify the complete ML workflow E2E tests follow representative input data through the complete training pipeline and inspect real outputs. Scope: Orchestrated dependencies from ingestion through validation, features, model fitting, evaluation and artifact/registry publication. Dependencies: Real or faithful staging equivalents of storage, task execution, tracking and registry integrations. Isolation: Dedicated test paths, databases, model names and resource quotas. Never let a test publish or deploy a production model accidentally. Speed: Generally the slowest and costliest layer; a full model train may require substantial CPU/GPU time. Failure diagnosis: A failed final artifact reveals a workflow problem but often requires tracing logs across multiple stages. Best placement: Pre release staging, scheduled smoke tests and selected pull request paths using reduced workload. A good E2E acceptance scenario verifies: The DAG schedules tasks in the intended order and propagates failures. Input snapshots and model artifacts have the correct run/partition identifiers. The model loads successfully and produces a finite prediction on a valid sample request. Evaluation results and metadata are linked to the correct training run. Retries do not duplicate dataset outputs, registry entries or other side effects. On failed steps, unapproved artifacts are not mistakenly published. A green E2E test does not prove the model is accurate on real world traffic; model performance evaluation and monitoring remain necessary. A numerical test suite example Assume a team has: 150 unit tests , averaging 0.02 seconds each → about 3 seconds sequentially. 20 integration tests , averaging 3 seconds each → about 60 seconds sequentially. 2 E2E smoke tests , each 8 minutes → about 16 minutes sequentially. Sequential total = 3 + 60 + 960 = 1,023 seconds, about 17 minutes and 3 seconds . If the team runs only unit tests before every local commit, engineers get rapid feedback. On pull requests they run all unit and selected integration tests. Before a release, run the E2E smoke suite in staging. These values are hypothetical. Test execution can be parallel, and setup/teardown overhead or infrastructure startup may dominate; do not assume the arithmetic predicts actual CI wall clock time. The point is to invest heavily in fast tests and reserve expensive E2E coverage for high value paths. What should be mocked—and what should be real? For unit tests: Mock API responses, object store reads, clock dependencies or expensive services when testing pure business logic around them. Use a stable fixture or local in memory object for inputs. Avoid mocking away the transformation being checked; an assertion against another mock's return value proves very little. For integration tests: Use a real test database to test SQL behavior. Exercise real serializer/deserializer and storage protocols where feasible. Replace external payment/notification services with approved test doubles unless their live integration is the subject of the test. Use distinct isolated resources and clean them up reliably. For E2E tests: Keep the complete workflow and critical interfaces real in staging where feasible. Reduce data volume and training epochs to contain cost, without quietly bypassing required DAG stages. Use synthetic or appropriately controlled non sensitive data and prevent production side effects. How tests fit into CI/CD An efficient sequence is: 1. Developer feedback: Linting, type checks, unit tests and DAG import/structure checks. 2. Pull request: All quick unit tests and targeted integration tests for changed components. 3. Merge or release candidate: Broader integration coverage and artifact/registry checks. 4. Staging: E2E DAG smoke run with controlled data and bounded resources. 5. Production deployment: Lightweight post deployment health checks; never rerun a destructive full training workflow merely to test service health. Failures should retain logs and artifact links, but test results must not leak access credentials or training data secrets. Diagnose failures by test layer Unit test fails: Inspect the smallest function or assertion first. A failed meters to km test points to transformation behavior, not the DAG scheduler. Integration test fails while unit tests pass: Inspect database migrations, authentication, network setup, serialization, schemas, permission scopes and real service behavior. E2E test fails while lower layers pass: Inspect dependency edges, scheduling, distributed artifact passing, task retries, environment configuration, data handoffs and registry publication. A broad E2E failure might reveal where the problem appears , not necessarily where it originated. Correlate task/run IDs across services to find the root cause. Important Airflow specific lessons Apache Airflow's current best practices guide distinguishes: DAG loader tests that detect import/configuration errors. Unit tests for operators, sensors and DAG structure. Real DAG runs , including dag.test(), for integration level behavior. Staging runs for complete workflow behavior. Airflow tasks can execute on separate workers or machines. A test that passes when two tasks share a temporary local file may fail in distributed deployment. Share large intermediate artifacts through supported durable storage and pass references or small metadata between tasks. Test this handoff during integration/E2E runs. Airflow also recommends idempotent tasks and explicit partitions; exercising a repeated DAG task or rerun should not create duplicate records. Trade offs and common mistakes Too many E2E tests: Slow, expensive CI with noisy failures and weak fault isolation. Only unit tests: Misses authentication, serialization, storage and orchestrator wiring problems. Mocking every service: Tests may validate the mocks rather than the real system. Testing against production data paths: Risks accidental writes or sensitive data exposure. Using arbitrary random test inputs: Makes failures harder to reproduce; use fixed, controlled cases. Skipping cleanup: Causes false failures from previous test artifacts. Running full size training as every E2E smoke test: Inflates compute expense without proportionate confidence. Calling a DAG import check an E2E test: It proves the DAG can be loaded, not that tasks execute correctly. Ignoring retries and distributed execution: A pipeline may pass locally but fail when workers do not share filesystems. Conflating test layer with enforcement severity: Even a tiny unit test can be release critical; Question 103 covers when to block execution. In short: Use many fast isolated unit tests , focused real boundary integration tests , and a few representative full DAG end to end tests . Each layer offers different confidence, speed and diagnostic precision. Make the test environment realistic where needed without sacrificing isolation, reproducibility or safety.
How do you test pipeline components with small deterministic fixtures?
A deterministic fixture is a tiny, known input dataset or controlled dependency whose expected result is fixed before a test runs. It lets an engineer detect whether an ML pipeline component changed behavior, without relying on live datasets, moving clocks, arbitrary randomness or fragile infrastructure. Question 104 explained unit versus integration versus end to end testing . Here the focus is designing and using small, reproducible test inputs and exact expected outputs inside those tests. Beginner friendly example: delivery ETA features A feature component accepts delivery records containing distance m and duration min. It produces: distance km = distance m / 1000. speed kmh = distance km / (duration min / 60) when duration is positive. An explicit validation error for zero or negative duration. Choose a three row fixture with results easy to verify manually: Order A: 2,000 m , 10 min → 2 km , 12 km/h . Order B: 1,500 m , 5 min → 1.5 km , 18 km/h . Order C: 0 m , 15 min → 0 km , 0 km/h . For Order A, 2 / (10/60) = 12 km/h. For Order B, 1.5 / (5/60) = 18 km/h. These expected values should be specified independently , not computed using the exact same transformation under test. If someone accidentally removes the /60 time unit conversion, the result for A becomes 2/10=0.2 instead of 12 . The fixture catches the error immediately. Build a small pytest fixture These tests use literal expected outputs and avoid a network connection. They expose calculation defects and boundary handling directly. In a production component, also validate string/numeric types, booleans, NaN and infinity, and any maximum physical limits. Why the fixture must be small and deliberate A 3 row example makes it possible to calculate and inspect each expected result by hand. A 10 million row production extract makes mismatches hard to diagnose and may be slow or privacy sensitive. A good fixture intentionally includes: A normal case with a meaningful result. A boundary case , such as distance zero. An invalid case , such as negative distance or zero duration. A missing value case if the actual contract permits null inputs or requires an error. An entity/ordering case where joins, duplicates or sorting matter. A tiny fixture is not intended to approximate the complete production distribution. Maintain separate realistic samples, property based tests and stress/performance tests for broader coverage. Keep expected values independent A flawed test can accidentally reproduce the same bug: That comparison proves essentially nothing. Better choices: Use hand calculated golden outputs. Compare with a separately reviewed specification or independent reference implementation. Assert important invariants, such as nonnegative speed for valid input. Use pytest.approx only when approximate numeric equality is appropriate. Make assertions informative: compare row counts, identifiers, types, expected values, and exception behavior rather than asserting that a function merely returned a nonempty object. Control randomness with a fixed seed ML components often shuffle rows, split train/test data, sample examples or initialize model parameters randomly. For a reproducible fixture test, use an explicit local generator : The test checks stable output across repeated runs of the same implementation/environment , not that every NumPy release or parallel architecture must always produce identical output. For scikit learn estimators and splitters, set and record random state according to the desired behavior. A fixed integer often gives repeatable estimator fitting/splits in the same setting; for scientific robustness, evaluate across multiple seeds rather than claiming a single fixed seed proves generalization. Important: A seed alone cannot guarantee byte for byte equivalence across different GPU kernels, multithreaded schedules, framework versions, hardware, and nondeterministic operations. Pin environments and use deterministic algorithm settings when required, and distinguish deterministic component tests from reproducible full training experiments . Make the clock deterministic A feature might calculate age minutes as current time − event timestamp . Calling datetime.now() inside a test makes the expected value drift. Inject the clock instead: The age is exactly 15 minutes every run. Use timezone aware timestamps and explicitly check that event timestamps are not unexpectedly in the future. An injected clock is usually easier to reason about than globally patching the wall clock. Isolate filesystem and external dependencies pytest's built in tmp path fixture provides a unique temporary directory for each test function: The test cannot overwrite a real training dataset because it writes to an isolated temporary location. In a production suite, similarly replace external APIs, credentials, object stores and databases with controlled substitutes only where appropriate to the test's purpose . Prefer a function scoped fixture if it creates mutable lists, dictionaries, stateful clients or files. Avoid shared mutable global fixtures that one test can change before another test runs. Use monkeypatch for environment variables or external calls, allowing pytest to undo changes after the test. Clean up spawned processes and external resources reliably. Never put real personal data, access tokens or production credentials in fixture files. Edge cases and repeatability checks Use @pytest.mark.parametrize to avoid copying nearly identical tests. Consider: Empty dataset and a single row dataset. Null values and missing required columns. Duplicate entity IDs and unsorted timestamps. Zero, negative, very large and non finite numbers. Daylight saving transitions and timezone conversions where relevant. Constant valued features, unseen categories and labels with only one class. Row reordering that should not change an order invariant aggregate. For an aggregation component, a tiny fixture with the same two orders in reversed sequence should yield an identical sum or mean. Do not assert identical results for operations whose contract explicitly depends on order. Failure diagnosis and coverage When a small fixture fails: 1. Read the exact input row and expected result. 2. Find the earliest transformation where output diverges. 3. Check the unit, formula, boundary rule and versioned dependency. 4. Add a regression fixture capturing the bug if it represents a new valid scenario. 5. Fix the implementation, not the expected result, unless the specification itself was wrong and approved for revision. Coverage should track important behavior and branches , not just the number of lines executed. A high line coverage percentage does not prove that null handling, units, data leakage, rare categories or feature semantics are correct. Fixture lifecycle and maintenance Store fixture code/data in version control next to the component tests. Use clear names such as orders two valid one zero distance. Document what behavior each fixture is testing. Change expected outputs only after reviewing a legitimate specification change. Keep test artifacts small, machine independent and free of sensitive data. Run deterministic component tests in every pull request. Periodically add fixtures derived from sanitized production incidents to prevent regression, while preserving privacy. Retain a small set of strict golden values; do not turn every incidental output into a brittle snapshot. Limitations Tiny fixtures will not reveal scale dependent failures, data skew or rare distribution tails. A fixed seed masks variability rather than demonstrating statistical robustness. Exact float equality can be brittle across different numeric backends; use explicitly chosen tolerances where reasonable. Mocks can hide real API or serialization mismatches. A deterministic test may pass while the deployed system still suffers from timing, concurrency or dependency failures. A test fixture can become outdated when business semantics evolve, so its expected values need ownership and review. In short: Use tiny controlled inputs, independently calculated expected outputs, fixed random generators, injected clocks, isolated storage and explicit edge case assertions . These make pipeline component failures quick to reproduce and explain, while broader integration, scale and model quality tests provide complementary confidence.
How do you run reproducible hyperparameter searches in an orchestrated pipeline?
A reproducible hyperparameter search is a managed set of training trials where each trial's data, model configuration, seed, code, environment and evaluation metric can be traced and re run. An orchestrator schedules the stages and resources; a tuning engine suggests trial configurations; an experiment tracker records their evidence. A fixed seed is useful, but it is not enough to guarantee exact reproducibility across parallel workers. Unlike Question 105, which focused on tiny deterministic test fixtures , this lesson concerns real hyperparameter optimization runs and reproducible orchestration . A beginner friendly delivery time model example A food delivery company trains a random forest regressor to predict delivery duration. Its pipeline uses: A frozen, versioned snapshot containing 20,000 delivery orders . A fixed split of 16,000 training rows and 4,000 validation rows , identified by immutable row IDs or saved split indices. The same evaluation metric for all trials: mean absolute error (MAE), minutes . Lower is better. A search space for max depth of 4, 8 or 12 , and min samples leaf of 2 or 5 . A complete grid contains 3 × 2 = 6 trial configurations. Assume the six validation results are: Depth 4, leaf 2 → 5.1 min MAE . Depth 4, leaf 5 → 5.3 min MAE . Depth 8, leaf 2 → 4.6 min MAE . Depth 8, leaf 5 → 4.8 min MAE . Depth 12, leaf 2 → 4.7 min MAE . Depth 12, leaf 5 → 4.9 min MAE . The best observed candidate is depth 8, leaf 2 , with 4.6 minutes MAE . This is an illustrative experiment, not a claim about real ETA datasets. The gap from the 5.1 minute baseline is 5.1 4.6 = 0.5 minutes, a relative MAE improvement of 0.5 / 5.1 × 100 ≈ 9.8%. This comparison is meaningful only if both candidates used exactly the same holdout and metric definition. What must be frozen before searching? 1. Dataset identity: Source snapshot or table revision, data checksum/manifest, label definition and feature calculation revision. 2. Splits: Stored train/validation row IDs or a reproducible splitting procedure and seed. For time series data, use a time respecting split rather than randomly mixing future observations into training. 3. Search definition: Parameter names, distributions/ranges, grid or sampling method, optimization direction and maximum trials. 4. Training code: Git commit, model implementation, preprocessing and feature selection logic. 5. Environment: Python and library versions, container digest, CPU/GPU configuration where relevant. 6. Randomness: Distinguish the search sampler seed from the individual trial's training seed ; persist both. 7. Evaluation: Same holdout, folds, metric, missing value handling and stopping rules for comparable trial scores. For fairness, never let each trial independently generate a different validation split unless the experiment explicitly evaluates that variance. Keep the final test set untouched until tuning and model selection finish. A practical orchestration layout An orchestrator such as Airflow, Prefect or Dagster can manage dataset readiness, job dependencies, timeouts and retries; Optuna or Ray Tune can manage suggestions, study/trial state, scheduling or checkpointing. MLflow can track trials as parent and nested child runs. Separate infrastructure scheduling from the algorithm's optimization state to make failures easier to recover. Concrete Optuna + scikit learn example The code below performs a sequential, persistent study with a fixed data split and explicit random seeds. Replace the demonstration dataset with a versioned, approved feature/label snapshot in production. How to interpret this example: The six trials are six TPE sampled trials , not necessarily the six unique combinations in the earlier six configuration grid. TPE may choose the same combination more than once. Use an actual grid sampler when you need each distinct grid combination visited. A local SQLite study is suitable for a simple sequential demonstration, not a recommended multi node shared storage backend . Use an appropriate central transactional database and lock safe tuning infrastructure for distributed workers. Also, load if exists=True resumes an existing named study; it does not guarantee that its stored sampler, objective, data or search space matches your current code. Store a study manifest and fail if incompatible before adding more trials. Compare trials with an experiment tracker MLflow supports nested child runs for hyperparameter tuning. For each trial, log: Study ID and trial ID, with parent run relationship. Sampled configuration and explicit trial seed. Data snapshot, split identifier, code commit and container version. Validation metric, units and direction. Start/end time, status, attempt ID, compute cost and worker type. Model artifact and/or checkpoint when relevant. Log the best eligible trial's ID to the parent study run so selection remains auditable. A simple logging pattern inside an existing parent MLflow run is: This snippet assumes mlflow, trial and score are defined, and that a parent run is active. In multiworker settings, link trials to the correct experiment/parent through explicit run identifiers; do not rely on thread local nested run state crossing processes. Parallel trials: throughput versus identical results Suppose each trial requires 12 minutes , and the budget allows 6 trials . Sequential execution: 6 × 12 = 72 minutes. Three independent workers, two waves: idealized 2 × 12 = 24 minutes. Idealized speedup: 72 / 24 = 3×. Actual wall clock time includes startup, data loading, queues, stragglers and resource contention. More parallelism may exhaust GPU/CPU or data store quotas. Reproducibility caveat: Adaptive algorithms such as TPE or Bayesian optimization may choose later configurations based on earlier completed trials. With parallel execution, trials finish in changing orders, so even a fixed search seed may lead to different future suggestions and a different winning configuration. Optuna explicitly recommends sequential execution for strongest reproducibility; Ray Tune also documents nondeterminism from trial completion order. For high throughput, aim for traceable, replayable trials rather than promise identical overall search trajectories. <See official Optuna and Ray references. Checkpointing and resuming searches There are two different states to protect: Study level state Trial identifiers and configurations, completion/failure states, objective values, sampler/scheduler history and search budgets. A persistent Optuna study or Ray Tune experiment storage enables a coordinator to resume without forgetting completed trials. Trial level state Model weights, optimizer state, epoch number, relevant RNG state and training progress. Needed if a partially trained model should resume rather than restart from epoch zero. For example, an experiment plans 20 trials , completes 12 , and then the worker coordinator crashes. After restoring a compatible study and durable artifacts, it should schedule 8 remaining trials , not blindly launch another 20. If 2 of the 12 are still marked incomplete or failed, the recovery policy must explicitly determine whether to retry those trials and how they count toward the budget. Ray Tune's persistent storage and Tuner.restore(...) help recover experiment state and optionally resume errored trials from their last checkpoints where supported. A study database alone does not recover a trial's model/optimizer state unless the training code actually saves and reloads it. Similarly, an interrupted Optuna trial may need to be retried as a new attempt under a tracked policy. Idempotent retry and consistent comparison Use stable study id + trial id + attempt id identifiers to prevent duplicate results. Avoid writing to one shared model filename from competing workers; give each trial an isolated artifact path. Distinguish retrying a failed trial from sampling a new configuration . Report failed/pruned/cancelled trials separately from successfully scored trials. Do not select a trial that failed evaluation or whose metric is missing/NaN. Treat a change in metric calculation, validation split or feature definition as a new comparison cohort, not a silent continuation of the old study. When checkpointing, record compatibility of code, dependencies and optimizer state before restoring across environments. If trial 7 reports 4.4 min MAE using a different validation set from all other trials, it is not automatically better than trial 3 at 4.6 min MAE . Normalize evaluation conditions before comparing. Model selection without overfitting the evaluation A search optimizes the validation metric. Repeatedly testing many configurations on the same holdout can overfit selection to it. Keep a separate held out test dataset or use well designed cross validation/nested validation where appropriate. After selecting the best configuration: 1. Refit on the approved training data (possibly including validation data, according to your evaluation protocol). 2. Evaluate once on the untouched test set. 3. Record model, data, feature and environment provenance. 4. Apply model acceptance and promotion gates; do not automatically deploy a model solely because it won a search. For a stochastic model, consider repeating the final configuration across multiple seeds to check score stability, while keeping the selection protocol explicit. Trade offs and limitations Grid search: Transparent and exhaustive for a small discrete space; grows exponentially as dimensions expand. Random search: Often efficient for larger spaces and easier to parallelize; record sampler state and seeds. Adaptive search: May find good configurations with fewer trials, but completion order effects complicate exact reproduction. Early stopping/pruning: Saves compute, but differing intermediate reports, resource contention and scheduler decisions can affect which trials finish. Checkpoints: Improve fault tolerance but consume storage and need compatibility/retention rules. Perfect determinism: Hard across GPU kernels, hardware, framework versions, multiprocessing and distributed scheduling. In short: Make each trial identifiable, comparable and replayable by pinning data and splits, saving configs and seeds, tracking code and metric versions, persisting study/trial state, and handling parallelism and retries deliberately. Reproducibility is an end to end operational contract, not a single random state=42 setting.
How do you manage compute budgets and quotas for parallel training runs?
Compute budget management for parallel training means controlling how many experiments run simultaneously, what CPU/GPU and memory each receives, how much they may cost, and when they must stop. A scheduler can enforce resource admission, but a resource quota is not a monetary spending cap : both controls are needed. Question 106 explained reproducible hyperparameter searches . Here the focus is the operational envelope for any set of concurrent training jobs: capacity, fairness, cost, scheduling, protection and recovery. Beginner friendly example: delivery time models A delivery company wants to run 12 training jobs . Its shared experimentation cluster has an available team allowance of: 4 GPUs . 16 CPUs . 64 GiB RAM . A project compute spend budget of $120 for this experiment batch. Each job requests 1 GPU, 4 CPUs and 12 GiB RAM and is expected to take 2 hours . The effective concurrency is constrained by all resources: GPU capacity: floor(4 / 1) = 4 jobs. CPU capacity: floor(16 / 4) = 4 jobs. Memory capacity: floor(64 / 12) = 5 jobs. Therefore the safe theoretical maximum is 4 concurrent jobs , assuming these are genuinely allocatable resources and leaving no extra headroom requirement. Twelve jobs require at least 3 waves , each roughly 2 hours, for an idealized 6 hour makespan. In practice, startup, other workloads, resource placement and differing runtimes increase this. Suppose each GPU machine costs $2 per GPU hour , with the example price assumed to include its attached CPU/RAM. Then the estimated compute cost is: 12 jobs × 1 GPU × 2 hours × $2/GPU hour = $48. The initial estimate leaves $72 of the $120 budget. It is not a complete bill: add storage, network, orchestration, idle worker charges, logging, failed attempts and other billable infrastructure as appropriate. Also check whether the provider bills whole instances rather than individual GPUs. Capacity versus budget: two different gates Capacity admission: Can the cluster accept the requested CPU, GPU, memory, node type and concurrency without displacing protected services? Budget admission: Is the forecast incremental cost within the project/team's remaining authorized spending envelope , including a reserve for retries and overhead? For example, 4 GPUs may be available, but launching another 20 hour job could violate the remaining project budget. Conversely, money may remain while GPU quotas or regional capacity prevent the job from starting. A useful preflight formula is: estimated run cost = estimated runtime hours × hourly compute rate + estimated overhead cost. Use a forecast with uncertainty and a safety margin, not just the most optimistic runtime estimate. Record estimated and actual costs per run, experiment, model and team. Set quotas at the right levels Organization / cloud account: Overall service quotas, allowed GPU families, regions, spending policies and reserved production capacity. Team / project: Monthly or experiment specific monetary budgets, maximum concurrent training jobs, and allocated GPU/CPU shares. Namespace / queue: Kubernetes ResourceQuota, object/pod counts, and fair scheduling so one experiment cannot consume the shared cluster. Individual job: Resource requests/limits, expected duration, hard deadline, retry policy, priority and output storage allowance. Kubernetes ResourceQuota constrains aggregate resource requests within a namespace, including requests.cpu, requests.memory and extended GPU resources such as requests.nvidia.com/gpu. Quotas don't guarantee physical capacity or fair scheduling by themselves. A workload can remain pending if the cluster lacks a suitable node, even when it is within namespace quota. Requests and limits are not interchangeable CPU request: Primarily informs scheduling and resource allocation under contention. CPU limit: Enforces a ceiling on CPU consumption, potentially throttling training and extending runtime. Memory request: Helps the scheduler select a node with sufficient capacity. Memory limit: Exceeding it can cause an out of memory kill. GPU allocation: Device plugin/extended resource requests are generally whole device allocations and are not overcommitted as ordinary CPU capacity; resource configuration and GPU sharing mechanisms vary. Job level deadline: Bounds how long a job is allowed to keep running, but it is not automatically a chargeback or budget mechanism. Requests should reflect observed need. Oversized requests waste schedulable capacity; too small requests increase throttling or OOM risks. Choose memory limits based on validated peak memory and planned headroom. An illustrative Kubernetes quota This must be created by someone authorized to administer the namespace. It is a namespace level allocation limit , not an instruction to allocate the full quota to each training job. If the actual Kubernetes cluster lacks the GPU device plugin/resource name, the GPU quota example requires adaptation. GPU devices requested by a pod also need appropriate per container resource declarations and compatible nodes. A training job might request the following: An actual Job needs this block inside its container configuration. With these illustrative requests, four pods use 16 requested CPUs and 48 GiB requested RAM , within the quota. However, the 16 GiB per container memory limits may be subject to separate limits.memory quota configuration. GPU request and limit must agree when specified. Concurrency controls in the orchestrator Limit both cluster resources and number of simultaneously active jobs. For example, Ray Tune allows tune.with resources() to declare per trial resources and TuneConfig(max concurrent trials=...) to set a concurrency ceiling: Here train model must be a defined, supported Ray Tune trainable. Ray scheduling reservations are not automatic operating system enforcement of all CPU usage or GPU memory: the training code must respect the requested resources. Other orchestrators can use parallelism limits, task pools, queues or Kubernetes admission policies. For independent jobs, favor a bounded queue and predictable fairness over starting every experiment at once. Give scheduled production workloads reserved capacity when production and research share infrastructure. Budget controls and alerts Soft warning threshold: For example alert at 70% and 90% of the $120 batch budget, corresponding to $84 and $108 . Admission threshold: Decline new experiment starts if their projected cost would exceed remaining budget plus a defined safety reserve. Stop policy: Cancel or preempt eligible low priority jobs when the approved project budget is at imminent risk; maintain enough bookkeeping to account for charges already incurred. Emergency override: Require a named approver, amount, scope and expiry. Cost reporting: Allocate spend by run ID, team, model, environment, GPU type and execution status. AWS Budgets can alert on actual and forecast spend and support budget actions, but its cost data is not real time . The service's documented billing updates can lag for hours, so a budget notification alone cannot guarantee that experiments stop at exactly the cost cap. Pair cloud billing reports with a job level scheduler/admission ledger and provider controls. Timeouts, retries and preemption Long running or stuck experiments can consume large budgets. Set: A queue wait timeout to detect unschedulable tasks. A run time deadline , chosen from known model/training needs and approval policy. Limited retries only for recoverable transient errors, not indefinitely retrying OOM from an undersized model. Checkpoint intervals when a restart is worth the storage and CPU overhead. Cancellation cleanup for idle GPU instances, disks and orphaned services. A lower priority job may be preempted to make capacity available. Preemption can lose uncheckpointed computation, and a restart may consume additional budget. For example, if a 2 hour experiment saves a usable checkpoint every 30 minutes , preemption could lose up to roughly 30 minutes of training progress, ignoring checkpoint overhead and recovery time. The detailed mechanisms of distributed checkpoint/recovery belong to Question 108. Kubernetes scheduling priority can help protect more critical jobs, but does not inherently enforce a monetary cap or guarantee that a newly preferred job can start immediately. Cost monitoring and diagnosis Track metrics per experiment and aggregated by team: Requested, allocated and observed CPU/GPU hours , with effective utilization. Peak memory, GPU memory use, OOM events and CPU throttling. Queue time, start delay, active duration, retries, preemptions and idle reservations. Estimated cost before launch, accrued cost during execution and final reconciled provider cost. Remaining project/monthly budget and forecast overspend. Cost per successful training run or useful experiment result. Quota rejections, pending jobs and unschedulable node types. A frequent mistake is to minimize raw GPU hours even if the cheaper configuration takes so much longer that total cost or delivery latency increases. Compare cost to quality and time to result , not only peak GPU utilization. Scheduling policies and trade offs Small CPU only baselines: Often cheap and useful for early experiments; don't allocate GPUs without evidence of benefit. One GPU per job: Simpler isolation and forecasting but less parallelism when GPU capacity is tight. Larger distributed jobs: Can reduce elapsed training time but increase scheduling complexity, communication overhead and simultaneous resource demand. Spot/preemptible instances: Lower list price compute in some clouds but higher interruption risk; suited to retryable, checkpointed tasks and not always to strict deadlines. Priority and fair sharing: Protects critical deadlines and avoids a noisy neighbor team monopolizing resources; implementing priority can increase wait times for low priority experiments. Aggressive memory/CPU limits: Can improve resource utilization but cause OOM or throttling; measure actual resource profiles rather than guessing. Over reserving: Raises idle cost and reduces availability for other teams. Under reserving: Causes failures, retries and hidden compute waste. Common mistakes Treating a Kubernetes GPU quota as a dollar budget. Launching more concurrent trials than the usable CPU/GPU and memory capacity permits. Assuming cloud budget alerts hard stop new jobs in real time. Ignoring failed attempts, storage and idle worker costs in the forecast. Giving every experimental task production level scheduling priority. Keeping GPU instances allocated while waiting for upstream data. Automatically retrying deterministic OOM failures without changing inputs or capacity. Setting CPU limits so low that throttling increases run time and total cost. Assuming a preempted run resumes from the same point without a tested checkpoint. Measuring only resource utilization rather than meaningful experiment outcomes. In short: Combine resource quotas, admission control, bounded concurrency, job deadlines, priority policies, cost monitoring and explicit budget approval . Resource schedulers keep the cluster safe; spending controls keep the experiments affordable; observation and recovery keep both policies realistic.
How do distributed training jobs checkpoint and recover after failure?
Distributed training checkpointing preserves a consistent snapshot of training progress so a failed worker, interrupted node, or restarted job can continue from a known good step instead of starting over. For faithful recovery, save more than model weights: optimizer and scheduler state, progress counters, random number generator (RNG) state, and the relevant data sampling position must be preserved or reconstructed. Question 107 covered resource budgets, quotas and preemption from the scheduling perspective. This lesson focuses on the technical checkpoint format, coordination, durable publication and restoration . Beginner friendly numerical example Suppose a delivery ETA model trains across 4 GPUs for 10,000 optimizer steps . Each step takes 0.5 seconds on average, excluding saving overhead. At step 4,000 , the workers successfully commit checkpoint ckpt 4000. At step 4,650 , one worker fails before the next successful checkpoint. After coordinated restart, the job loads the last fully committed step 4,000 . Lost progress = 4650 4000 = 650 steps. Recompute time ≈ 650 × 0.5 = 325 seconds, about 5 minutes 25 seconds , before restart overhead. Never resume at 4,650 just because logs report reaching that step: there is no verified checkpoint for the model and optimizer state at 4,650. If checkpointing occurs every 1,000 steps , maximum lost progress from a failure immediately before the next save is approximately 999 steps . At 0.5 seconds per step, that is about 8 minutes 20 seconds . Actual loss also depends on checkpoint timing and whether asynchronous writes have completed. What is inside a resumable checkpoint? Model parameters: Weights and buffers, including sharded tensors in FSDP/ZeRO style training. Optimizer state: Momentum, Adam running averages and other parameter specific variables. Reloading weights without optimizer state often changes the next training updates. Scheduler and scaler: Learning rate scheduler position and mixed precision gradient scaler state, if used. Training progress: Completed epoch, global optimizer step, accumulation position, sample count and any stage specific state. Data loader/sampler state: Current shard, shuffled permutation/seed, batch offset or equivalent replay position. Otherwise a restarted run can repeat or skip examples. RNG state: Python random, NumPy, torch CPU and the appropriate CUDA/device RNG states, preferably per rank. Capture these consistently with training step boundaries. Run identity: Dataset snapshot, training configuration, software/container version, distributed topology, world size and model/optimizer checkpoint format. Integrity metadata: Checkpoint ID, completion manifest, expected shard list, checksums and committed timestamp. A stored directory is not enough evidence that all rank shards are valid. For gradient accumulation interrupted midstep, either preserve gradient accumulation state or checkpoint only at a clearly defined optimizer step boundary and replay from that boundary. Saving partial gradients from some workers but not others is not a consistent recovery point. Coordinated snapshots across workers Imagine four ranks: 0, 1, 2 and 3. In sharded training, each rank may own different model or optimizer shards. A usable checkpoint must correspond to the same logical step across all ranks. 1. Finish the target optimizer step and reach a coordinated checkpoint boundary. 2. Construct state dictionaries with the supported distributed checkpointing APIs. 3. Have participating ranks write their shards to a new, unique checkpoint location. 4. Validate that the distributed save finished across the participating ranks. 5. Publish an atomic or transactional commit marker / manifest only when all required pieces are durable and verified. 6. Advance the latest committed pointer to the new checkpoint. 7. Keep the previous committed checkpoint available until the new one has passed verification. PyTorch Distributed Checkpoint (DCP) supports parallel saving/loading and resharding on load. DeepSpeed explicitly warns that save checkpoint must be called by all participating processes , not just rank 0, or the operation can hang due to synchronization. <See the linked PyTorch and DeepSpeed references. Do not mix shards from different steps. An apparently complete directory containing rank 0 state from step 5,000 and rank 1 state from step 4,000 is not a valid checkpoint. Atomic and durable writes A safe publication protocol looks like: In a POSIX compatible filesystem, an atomic rename can help publish a completed checkpoint within the same filesystem, subject to durability guarantees . In object stores, a multi object upload is not an atomic directory rename; write immutable objects and publish a small commit/manifest object only after validation. Readers should resolve the committed manifest rather than scanning for the newest looking prefix. Design for power loss and worker failure during upload. If ckpt 5000 is incomplete, the loader must reject it and fall back to the last committed checkpoint, for example ckpt 4000. Checksums detect corruption; they do not by themselves guarantee that shards came from one consistent optimizer step. Record both identities and integrity. PyTorch Distributed Checkpoint example A compact conceptual pattern using modern PyTorch DCP is: Important limitations: This demonstrates the DCP entrypoints, not a ready to run FSDP or ZeRO checkpoint recipe. Correct sharded state dictionaries should use APIs appropriate to the installed torch version and model topology. This snippet omits scheduler, scaler, RNG, sampler, progress, error handling, shared storage configuration, and a separate committed manifest protocol. Follow the official DCP tutorial for real distributed setup, especially if resharding to a different world size. For a single process experiment, ordinary torch.save with appropriate durability handling may be enough. For sharded large models, use DCP or the distributed framework's native save/load mechanism. Recovery after a worker failure 1. Detect failure: Collective communication error, node termination, unhealthy rank or scheduler notification. 2. Stop or coordinate surviving workers: Do not let some ranks continue while others restore an older optimizer step. 3. Select checkpoint: Resolve latest committed; verify metadata, shard checksums and format compatibility. 4. Reconstruct topology: Restart the required ranks/process group and allocate compatible devices. 5. Load state: Restore model, optimizer, scheduler and scaler along with RNG and data position. 6. Verify progress: Ensure all ranks agree on global step and training configuration. 7. Resume at a safe boundary: Continue from the next step after the last committed optimizer update. 8. Validate recovery: Compare short resumed runs with an uninterrupted baseline where determinism is expected; monitor for duplicate data and divergent metrics. An elastic restart can have a different number of workers. PyTorch DCP can reshard checkpoint state on load, but resharding alone does not preserve identical data order or optimizer semantics when global batch size, sharding or learning rate policy changes. Specify supported topology changes and test them. RNG state restoration Restoring weights is not the same as restoring the random stream. Example RNG capture pattern: This is an example of what to capture , not a complete serialization or distributed restoration implementation. Capture rank specific RNG state and restore it after the process topology and generators are initialized , avoiding unintended random draws before training resumes. Modern NumPy code may use independent Generator instances whose bit generator.state must be handled separately. Sampler and data loader workers also require coordinated seeds/positions. Exact bitwise replay may still be impossible across differing GPUs, nondeterministic kernels or framework versions. Distinguish recovering useful training progress from identical numerical replay . Choose a checkpoint frequency: numerical trade off Assume each full coordinated checkpoint adds 12 seconds of blocking save overhead, and failures are rare and roughly uniformly distributed within each interval. With a checkpoint every 20 minutes : Save overhead ≈ 12 seconds per 20 minutes = 1% of elapsed training time. Expected lost work after a failure ≈ 10 minutes , if failure timing is uniform. With a checkpoint every 5 minutes : Save overhead ≈ 12 / 300 × 100 = 4% . Expected lost work ≈ 2.5 minutes . More frequent saves reduce lost progress but consume additional I/O, storage and compute. The truly optimal interval depends on failure rate, save time, restart cost, checkpoint size and training speed; use measured values rather than fixed universal defaults. Asynchronous saving can overlap storage writes with training, reducing blocking time, but increases temporary CPU memory pressure and operational complexity. PyTorch's async DCP recipe recommends controlling in flight saves, often allowing just one outstanding asynchronous checkpoint at a time, and waiting for its upload to complete before declaring success. What to monitor in production Last committed checkpoint ID and global step—not simply last attempted save. Save duration, upload duration, total checkpoint bytes and validation failures. Time since latest valid checkpoint and estimated recoverable work at risk. Number of failed workers, coordinated restarts and recovery time. Resume success rate, checkpoint compatibility errors and corruption incidents. Sample/epoch replay anomalies and unexpected jumps in metrics after restart. Durable storage availability and capacity sufficient for incomplete plus prior valid snapshots. Fault injection tests should deliberately terminate one worker during training and another during a checkpoint write. Verify that the job restarts from the last complete checkpoint, does not mix shards, and can continue without silently discarding optimizer state. Common mistakes Saving only model weights and claiming identical optimizer resumption. Writing each rank's state independently without a coordinated step or completion protocol. Loading the newest directory even when one rank's upload failed. Letting rank 0 alone call a collective save API. Keeping checkpoints only on an ephemeral worker disk. Advancing latest before the durable write completes. Forgetting RNG state, data loader progress or gradient accumulation boundaries. Assuming a changed world size implies unchanged sample order and numerical results. Creating overlapping async checkpoints until CPU memory is exhausted. Never testing actual restore paths until the first production incident. In short: Make checkpoints complete, coordinated, immutable until committed, durable and restorable . Save model and optimizer state alongside progress, RNG and data position; restart all required workers consistently; reject partial writes; and tune checkpoint cadence against expected lost work and save overhead.
How do you manage training job outputs and artifact retention?
Training artifact management is the practice of storing every important output from an ML training run with a stable identity, traceable provenance, appropriate access permissions and an explicit retention or deletion policy. The objective is to make successful runs reproducible and recoverable without retaining every temporary file forever . Question 108 explained how distributed checkpoints are created and restored during failures . This lesson focuses on the broader lifecycle after an artifact is produced : where it lives, how it can be found, which artifacts are protected, when storage classes change and how cleanup happens safely. What does a training job produce? Final model artifacts: Serialized weights, model format, tokenizer/preprocessing components, inference signature, model environment and inference configuration. Intermediate checkpoints: Potential recovery states and step/epoch markers. Their retention can be shorter than a published model's retention. Evaluation artifacts: Metrics, confusion plots, holdout predictions, slice analysis, error reports and comparison evidence. Experiment evidence: Configurations, random seeds, code revision, runtime/container digest, training logs and resource/cost summaries. Data/feature references: Immutable dataset snapshot IDs, feature view versions and split definitions. Prefer pointers/checksums over copying large datasets into each run when the underlying snapshots have an appropriate lifecycle guarantee. Temporary outputs: Scratch files, cache, intermediate tensors and staging uploads. These should be short lived, not confused with durable reproducibility artifacts. A metrics value stored in an experiment tracker is often metadata , while a model binary or plot is an artifact . MLflow documents this separation: its tracking backend stores parameters, metrics and tags; its artifact store holds potentially large files. Practical example: organizing delivery time experiments A food delivery company trains a new ETA model every day. It uses a structured object storage path: This is an illustrative storage layout , not an actual bucket or a required MLflow path. Use storage provider object version IDs or content hashes to distinguish immutable revisions. Make run IDs globally unique within the required scope and avoid overwriting old successful artifacts. An immutable manifest might record: run id, experiment id, model name and execution timestamp. Git commit and container image digest. Dataset snapshot ID, approved split/feature definition revisions and label specification. Training configuration, seed and dependency lockfile reference. Model and checkpoint artifact URIs, sizes, checksums and creation timestamps. Validation metrics, evaluation report URI and artifact format. Parent/child trial relationship, registry model/version and deployment references. Retention class, owner, protection status and deletion eligibility. The storage URI alone is a location , not a guarantee that content remains immutable; prefer immutable or versioned objects and checksum verification. Why lineage matters Suppose an incident occurs after ETA model v18 is deployed. Its registry entry points to a final model artifact, which points to training run run 042. The run manifest links to snapshot orders 20261102 a, code commit abc123 and feature definition eta features v4. This chain answers: 1. Which exact model produced the prediction? 2. Which dataset and feature definitions were used to train it? 3. Which code and environment should recreate it? 4. Which tests and evaluation evidence approved it? 5. Which artifacts must not be deleted while this model is serving or available for rollback? Maintain bidirectional references where practical: a model should locate its run, and retention tooling should be able to identify all active registered models that rely on an artifact. A registry reference to a deleted model binary is not a useful rollback strategy. Numerical storage cost and retention example Assume the team runs 30 experiments each month . Each run produces: One 2 GiB final model . Five 2 GiB checkpoints . 0.5 GiB of reports and logs . Total per run: 2 + (5 × 2) + 0.5 = 12.5 GiB. If everything is retained: Monthly growth = 30 × 12.5 = 375 GiB. Twelve months of runs = 4,500 GiB , approximately 4.39 TiB . Now define a hypothetical tiered policy: Retain all final models and reports for 12 months for reproducibility and review. Retain all intermediate checkpoints for 30 days unless a run is explicitly protected. Keep registered production and rollback model versions for as long as they are referenced , with a separate reviewed release retention rule. Transition eligible older artifacts to colder storage if retrieval time and minimum duration costs are acceptable. Ignoring calendar month boundary effects, the approximate steady state storage after 12 months would be: Final models plus reports: 30 × 12 × 2.5 = 900 GiB. Last month of checkpoints: 30 × 10 = 300 GiB. Total ≈ 1,200 GiB rather than 4,500 GiB . Estimated reduction = 4,500 1,200 = 3,300 GiB, roughly 73.3% less retained storage . This simplified estimate assumes every final model and report is retained, no extra replicas, versioned objects, incomplete uploads or protected checkpoints, and no per object pricing overhead. At an illustrative flat $0.023 per GiB month , storage cost would be roughly: Retain everything: 4,500 × $0.023 = $103.50/month. Tiered retention: 1,200 × $0.023 = $27.60/month. Illustrative steady state difference: $75.90/month . The rate is not a current vendor price quote . Real costs differ by provider, region, storage class, object count, retrieval charges, minimum storage duration, request operations, network transfer and replication. Do not treat a cold tier migration as free savings. Define retention by artifact role, not by age alone Scratch/temp artifacts: Delete after the pipeline succeeds and any required debugging grace period ends; clean abandoned multipart uploads and incomplete staging paths. Intermediate checkpoints: Keep a limited count and recent window for active/restartable runs; protect a checkpoint while a job is still running or recoverable. Completed experiment diagnostics: Keep enough logs, configs and evaluation evidence to debug and compare runs under agreed internal retention. Registered and deployed models: Preserve model binaries and the necessary compatibility metadata while production, shadow deployment or rollback depends on them. Regulated or legal hold artifacts: Follow approved retention and deletion requirements, including relevant legal or organizational policy; they may override default cleanup. Training inputs: Snapshot retention must be coordinated with model reproducibility goals. A training manifest pointing to a dataset already expired by another lifecycle rule is not sufficient for exact reruns. These durations are business policies, not universal defaults . Define minimum retention, maximum retention where applicable, legal holds, and owner approved exceptions. Manage outputs using MLflow MLflow Tracking logs run metadata and artifacts and records artifact URIs. A compact pattern: The example requires MLflow configuration and an existing outputs/evaluation.json. In a deployed system, use a shared durable artifact store rather than a worker local path; the proper MLflow artifact URI can be queried through its APIs. Model artifacts should be logged with a supported model flavor API and separately evaluated and registered as appropriate. Important deletion nuance: MLflow documentation states that deleting a run does not immediately remove all backend metadata and corresponding artifact objects, because recovery is supported. Its mlflow gc command can permanently remove eligible deleted runs and artifacts. Do not assume delete run instantly reclaims storage. Production cleanup requires carefully coordinated metadata and object deletion. Lifecycle policies and safe cleanup automation Object storage providers such as Amazon S3 and Google Cloud Storage support age based lifecycle rules for transitioning or deleting objects. However, bucket wide age only deletion can remove files still referenced by active models. A safer design is: 1. Inventory artifacts: Enumerate run IDs, object versions, total bytes and lifecycle class. 2. Compute reachability: Find artifacts referenced by production, approved, active training, rollback candidates and policy holds. 3. Select eligible objects: Apply retention rules only to unprotected artifacts past their cutoff. 4. Dry run the deletion: Produce object/version IDs, sizes, affected runs and estimated cost/recovery impact. 5. Require review for sensitive deletion: Especially for production/rollback assets or data under retention restrictions. 6. Delete in a controlled batch: Apply object version–aware delete operations; record outcomes and errors. 7. Reconcile: Verify remaining registry references, storage size and deleted metadata records. 8. Retain audit evidence: Store cleanup manifest, approver, rule version, timestamp and exception details. Never use a mutable object prefix as the only proof that an artifact is unused. A deployed model might read from that prefix even if the experiment run is marked deleted. S3 versioning caveat: Expiring a current object in a versioned bucket may create a delete marker while older object versions remain billable. Noncurrent version expiration is a separate rule. Object Lock or unfinished replication can affect deletion eligibility. Google Cloud Storage retention policies and object holds also restrict lifecycle deletion. Validate actual provider semantics before enforcing cleanup. Access controls, encryption and artifact integrity Apply least privilege read/write/delete permissions per team and environment. Restrict production model writes and deletion more tightly than experimental outputs. Encrypt artifacts at rest and in transit, with keys managed under organizational policy. Avoid storing credentials or unnecessary personal data in logs, manifests or model metadata. Verify checksums/digests on critical artifact uploads and downloads. Separate deployment credentials from training job credentials. Audit downloads, permission changes and deletion of high value artifacts where supported. A model file can contain unsafe serialized code (for example, untrusted pickle objects); only load trusted artifacts using the chosen framework's documented safe loading practices. Storage access control protects both integrity and confidentiality. Recovery and disaster readiness checks A retention policy is safe only if it preserves what recovery workflows need. Test that: The last approved production model and its dependencies can be downloaded and loaded. A supported rollback version remains accessible. Needed checkpoint(s) for an interrupted run are within their retention window. The run's dataset snapshot, feature version and environment identifiers can still be resolved. Model integrity and associated evaluation evidence survive partial storage outages. Recovery from cold or archival storage meets operational RTO and RPO requirements. A colder storage tier may reduce ongoing cost but impose retrieval delay or minimum duration charges. Keep rapid rollback copies in appropriate storage when rapid recovery is a requirement. Monitoring storage and artifact retention Track: Stored GiB/TiB and monthly growth by project, artifact class and owner. Storage and retrieval spend by class and location. Number of orphaned artifacts (no valid references) and broken model URIs. Protected artifacts held longer than their nominal retention. Failed/incomplete uploads and abandoned staging objects. Cleanup dry run estimates, deletion outcomes and audit records. Oldest reproducible training run and verified rollback model availability. Time to retrieve a model from each storage tier. Review deletion failures as carefully as upload failures: retention errors may cause unexpectedly high cost, while overly broad cleanup can destroy reproducibility or production recovery options. Trade offs and mistakes to avoid Keep everything: Simplest operationally, but causes cost growth and possibly excessive data retention. Delete aggressively: Saves storage but risks breaking rollback, audit or model reproducibility. Cold tier all artifacts: Lowers storage rates but can make urgent restore slow or expensive. Deduplicate identical blobs: Can save space if artifacts are content addressed, but reference counting and garbage collection become essential. Version every overwrite: Helps auditability but may leave costly noncurrent versions unless the lifecycle is designed. Retain only the final model: Saves space but often loses evidence needed to explain how it was trained and approved. Rely on object age only: Can delete old yet actively deployed model versions. Assume metadata deletion removes files: Creates hidden storage growth in some tracking systems. Store everything on local worker disks: Artifacts can vanish when a container or compute node terminates. In short: Give each training output a stable identity, durable location, ownership, lineage and explicit retention class . Protect artifacts referenced by active models, automate reviewed cleanup for everything else, and regularly test that required recovery and reproducibility paths still work.
How do you design a model evaluation gate before registration?
A model evaluation gate is an automated, auditable decision made after training and before a candidate model is approved for registration. It checks whether the model was evaluated on trustworthy data, performs well enough against a meaningful baseline, meets important segment level requirements, and has the evidence needed for review. This is more specific than Question 103's general policy for blocking or warning tests. Question 109 covered storing artifacts; here the focus is deciding whether the trained artifact deserves approval . A model registry can also retain rejected experimental versions in some workflows, so define whether the gate blocks all registration or specifically approved/production eligible registration . This article uses the latter principle: rejected candidates may remain in experiment tracking but must not receive approved status. Beginner friendly example: delivery ETA model A delivery company currently uses ETA model v12 and has trained candidate v13 . Both must be scored on the same frozen, time based holdout of completed deliveries, representing future like conditions and excluded from training and hyperparameter tuning. The release policy, agreed before seeing candidate results, requires: Overall MAE: At most 4.5 minutes . Improvement over baseline: At least 0.2 minutes lower MAE than the currently approved model. Peak hour slice MAE: At most 6.0 minutes . Rural delivery slice MAE: At most 6.5 minutes with adequate sample size. Data integrity: No known target leakage or unauthorized data usage. Evaluation completeness: All required metrics, slice counts and provenance must exist. Suppose baseline v12 has 4.8 minute overall MAE , and v13 has 4.4 minutes . Absolute improvement = 4.8 4.4 = 0.4 minutes. Relative improvement = 0.4 / 4.8 × 100 ≈ 8.33%. Overall MAE 4.4 ≤ 4.5 : PASS. Baseline improvement 0.4 ≥ 0.2 : PASS. But if candidate rural MAE is 7.2 minutes , exceeding 6.5 , the candidate fails the approval gate , even though it wins overall. The result is REJECT / HOLD FOR INVESTIGATION , not an automatically registered production candidate. This example shows why a single aggregate score cannot decide whether a model is operationally acceptable. Training, validation and final holdout are different Training data: Used to fit model parameters. Validation data: Used to choose hyperparameters, features, early stopping and other development decisions. Final holdout/test data: Reserved for an unbiased final comparison after the model and evaluation policy have been fixed. In a time dependent system, use chronological and entity aware splitting appropriate to the data, rather than randomly mixing future labels or duplicate orders into earlier training rows. If many models are repeatedly tested against the same holdout and developers then tune to those results, the holdout gradually becomes part of model selection and loses its independence. Retire or refresh such holdouts using a governed evaluation protocol. For example, train on January–August, tune on September, and hold out October, subject to the event chronology, label delay and seasonal representation requirements. A prospective shadow period may add confidence after offline validation. Design the gate in distinct stages 1. Verify evidence identity Candidate run ID, model artifact checksum and training code revision. Dataset/label snapshot ID and evaluation holdout version. Feature definition versions and relevant preprocessing steps. Metric definitions, evaluation environment and test script commit. Baseline model ID and the reason it is appropriate for comparison. 2. Enforce evaluation data integrity Reject a missing, unapproved or contaminated holdout. Check exact sample count, label coverage, duplicate entities/events and leakage constraints. Ensure no post prediction fields, outcomes, or future aggregations entered inference features. Compare baseline and candidate on the same eligible examples and units. 3. Evaluate task metrics Choose a primary metric aligned with business loss, for example MAE in minutes for ETA. Use supporting metrics when relevant, such as P90 absolute error, bias and prediction latency. Define metric direction, thresholds, sample eligibility and treatment of missing predictions. Make the required metric set mandatory; an absent metric is not a pass. 4. Inspect meaningful slices Examples: rush hour, weather conditions, urban versus rural, short versus long trips. Record each slice's sample count, metric and confidence interval. Predeclare critical slices and maximum acceptable regressions. Avoid treating a tiny, noisy slice statistic as precise evidence. 5. Apply decision and record it Approve only if every mandatory gate passes and evidence is present. Otherwise hold/reject with failed rule IDs and observed values. Route exceptions for human review where appropriate; do not silently lower thresholds. Persist the decision, rule version and approval identity with the candidate run. Numerical uncertainty: don't overinterpret a tiny improvement Suppose baseline MAE is 4.50 minutes and candidate MAE is 4.46 minutes on the same holdout of 2,000 deliveries . The observed gain is only 0.04 minutes , or 2.4 seconds per order. If a paired bootstrap over delivery level loss differences estimates a 95% confidence interval for MAE improvement of [ 0.06, 0.14] minutes , zero lies inside the interval. The data are compatible with both a slight worsening and a modest improvement. A release policy requiring credible positive gain would hold the candidate, even if it technically scores lower MAE. A practical paired bootstrap procedure: 1. Compute each order's absolute error under baseline and candidate. 2. Form per order gain abs error baseline abs error candidate. 3. Resample eligible independent evaluation units with replacement , keeping the two models paired. 4. Calculate average gain for each bootstrap replicate. 5. Report the percentile interval and check the predefined decision criterion. If examples are correlated within restaurants, cities or days, use a suitable grouped/block resampling strategy; independently resampling every row can understate uncertainty. Intervals depend on sampling assumptions, and confidence intervals are not guarantees about production performance. Slice level checks and sample sizes Suppose a rural slice contains only 15 deliveries , while urban contains 1,800 . A large rural MAE may deserve investigation, but its uncertainty is much larger than the urban metric. The gate might require at least 100 evaluation examples per critical slice to make an automated threshold decision; if a critical slice is underpowered, report INSUFFICIENT EVIDENCE and require a planned data collection or review route rather than silently passing it. The numbers here are hypothetical policy examples, not universal minimum sample sizes. Determine sample size targets using expected variance, acceptable risk and desired sensitivity. Also assess whether model improvement on one slice hides serious harm on another, and ensure segment definitions are derived from permitted data in compliance with privacy rules. Leakage checks belong before approval A model with unrealistically good results can be leaking target information. For ETA prediction, avoid: The actual delivered at timestamp as an inference time input. A feature computed using the completed order's final duration. Rolling averages that include events occurring after the prediction time. Multiple versions of one order accidentally split between training and holdout. Validate feature availability at prediction time, reconstruct historical joins correctly and use entity/time overlap checks. A leakage failure should invalidate the evaluation even if MAE is excellent. Python gate example The following simple policy consumes already computed metrics and returns named reasons. It does not calculate the metrics or validate data provenance itself. Production code must also validate nonnegative counts, metric ranges, immutable evidence identity and any uncertainty requirements. A gate should execute in trusted orchestration with fail closed behavior when its own validation or evidence retrieval fails. MLflow integration MLflow's classic model evaluation supports mlflow.models.evaluate() and a dedicated mlflow.validate evaluation results() API with MetricThreshold. It can check absolute thresholds or minimum improvement compared with a baseline, using the correct metric keys. This is an illustrative validation call requiring actual EvaluationResult objects and the installed MLflow release's expected metric name. A passing MLflow threshold alone does not enforce rural/peak slices, holdout independence, statistical uncertainty or leakage; those require separate recorded checks. The validated candidate can be passed to a registration step only after all required checks succeed. Preserve an audit link to evaluation evidence in the registry version's metadata or related release record. How registration and rollback fit in A simplified safe pipeline: Registration is not deployment. A candidate passing offline evaluation still needs integration, loadability, safety, latency and live rollout gates appropriate to the organization. If the new production version degrades after deployment, roll back to the last independently validated compatible model/feature interface. Keep that rollback artifact and its dependency metadata accessible. Question 109 explains retention policy rather than release decision mechanics. Common mistakes Choosing an evaluation threshold after seeing the candidate's score. Comparing models trained or evaluated against different holdout definitions. Reusing a repeatedly tuned test set and claiming unbiased model quality. Optimizing overall MAE while ignoring critical underperforming slices. Accepting a tiny gain without checking variance or confidence. Treating missing or corrupted evaluation evidence as a success. Letting an evidently leaked model pass because its accuracy is high. Registering a candidate as approved before all gate outcomes are persisted. Automatically deploying a registered model without separate serving checks. Deleting an older model that is still required for rollback. In short: Build a versioned pre registration evaluation contract : trusted holdout, same data baseline comparison, predefined metric thresholds, slice and uncertainty checks, leakage checks and complete provenance. Make approval automatic only when every mandatory condition is satisfied; preserve clear rejection evidence and keep deployment approval separate.