MLOps

MLOps applies software engineering and operations discipline to machine learning. It covers reproducible training, model registries, deployment, monitoring, governance, and incident response.

How do you prevent stale or unapproved datasets from entering training?

Prevent stale or unapproved datasets from entering training by enforcing a dataset admission gate before any training job reads its inputs. The gate must resolve an immutable dataset snapshot , verify its approval and quality evidence , check the dataset is recent enough for this specific training use case , and record what it admitted. If required evidence is missing, fail closed. Question 110 evaluated a trained model before registry approval. This lesson evaluates the training input before model fitting. Earlier lessons explained online feature freshness (Question 94), upstream arrival delays (Question 102), and dataset quality tests (Question 101); the new concern is combining those signals into a binding, auditable admission decision that cannot be bypassed by switching to a different dataset path after approval. Beginner friendly example: delivery ETA retraining A food delivery team retrains an ETA model each morning at 08:00 UTC , using the most recent approved daily orders snapshot. Its policy requires: Dataset's latest included complete order date is at most 1 calendar day behind the required reporting date. Dataset status is exactly APPROVED; PENDING, REJECTED, REVOKED and missing status do not qualify. Quality evidence was recorded as PASS for the same immutable snapshot ID , with a supported validation policy version. The dataset came from an authorized, committed source and its checksum/manifest matches what the gate approves. Required legal, access and data usage approvals are valid for the planned training use. At 08:00 on November 6, the expected reporting date is November 5 (UTC). Three snapshots are available: Snapshot A: through November 5, approved, quality PASS, digest verified → ADMIT . Snapshot B: through November 5, quality PASS, but status PENDING → REJECT until approved. Snapshot C: through November 3, approved, quality PASS → REJECT as too old for this specific policy. The gate selects A by its version and digest , never by a mutable path such as orders/latest. Numerical example: freshness and approval are independent Suppose the gate uses an elapsed time bound of 36 hours for the newest observation in the approved training snapshot. Training starts on November 6 at 08:00 UTC . Snapshot A's coverage watermark is November 5 at 12:00 UTC → age 20 hours → freshness PASS. Snapshot C's coverage watermark is November 4 at 12:00 UTC → age 44 hours → freshness FAIL by 8 hours . Even a recent snapshot with a fresh 20 hour watermark must be refused if its approval status is PENDING. Conversely, a previously approved snapshot must not be assumed fresh forever. Important: A coverage watermark represents the most recent included business event or complete period; a created at timestamp only shows when someone built the files. Re exporting old data today does not make the underlying observations current. Define the clock and boundaries explicitly; an event time maximum by itself does not prove coverage completeness. Design a dataset admission manifest Store a signed or access controlled admission manifest alongside immutable dataset revisions. For example: These are illustrative fields , not a vendor specific manifest schema. In production, keep approval records in a trusted service rather than trusting arbitrary user editable metadata in a file. A stable ID identifies the logical version ; a content checksum or immutable object version reference binds that ID to specific bytes . Store and verify both. Admission workflow 1. Resolve the intended input: The pipeline requests delivery orders for a defined logical training date, policy version and purpose. 2. Discover candidates: Query a trusted dataset registry/catalog for committed immutable versions. 3. Validate the source and snapshot: Check object/table version, manifest integrity, authorized location, digest and expected coverage. 4. Verify approval: Resolve a current, authoritative approval record whose scope includes the dataset version and training purpose. 5. Verify quality evidence: Require a passing quality report for the exact snapshot and the supported quality policy; reject unknown or expired evidence. 6. Apply freshness cutoff: Compare the coverage/event time marker with the training run's fixed logical evaluation time and permitted maximum age. 7. Issue an admission decision: Return the exact version, digest, policy revision, approval and evidence IDs. 8. Bind the job to the decision: Pass an immutable URI/object version or snapshot ID to training; avoid a second lookup of latest. 9. Recheck at use time where necessary: Approval revocation, credential expiry or authorization changes may occur between gate and read. 10. Audit the run: Persist accepted or rejected candidate IDs, reasons, timestamps and actual consumed snapshot. The result should be a reproducible dataset receipt associated with the training run, not a boolean alone. A simple Python admission guard This is a policy illustration only : it does not cryptographically verify a digest, validate the actual permission scope or bind a quality report to the same snapshot. Production admission must fetch trusted approvals, quality reports and immutable storage identities from controlled services. Never permit a user to manufacture APPROVED by editing a JSON or YAML file. How an orchestrated pipeline enforces it For an Airflow, Dagster or similar workflow, place this admission step as a hard dependency immediately before data loading. Only emit the immutable dataset identifier after successful checks. Make the data loader reject mutable aliases and unsupported versions independently; a skipped gate should not provide an easy path around policy. Dagster's asset checks can monitor quality/freshness and prevent downstream asset materialization under the configured execution policy. But a green check still must be associated with the specific version consumed by the training job . Do not treat a green dashboard for yesterday's partition as approval for a different revision published today. Prevent time of check/time of use (TOCTOU) bugs Consider this dangerous sequence: At 08:00 , orders/latest points to approved snapshot A and the gate passes. At 08:01 , an upstream process repoints orders/latest to unapproved snapshot B. At 08:02 , the trainer resolves orders/latest again and reads B. The gate passed, but training consumed the wrong data. Fix: The admission result includes immutable snapshot id=A and the expected content digest; the training loader resolves that exact immutable version and verifies it. If object immutability cannot be guaranteed, use the storage system's snapshot isolation/versioning and validate the actual bytes read against approved identity. The trainer must record the version consumed, not just the version that was checked. Lineage and allowlists A training allowlist should be explicit about: Dataset names, storage accounts/buckets/tables and allowed source owners. Approved snapshot identities and required status. Permitted business purposes, environments and model families. Approved transformations and feature/label definitions. Allowed quality policy versions and evaluation windows. A path allowlist alone is insufficient: an allowed bucket can contain an unapproved dataset. Conversely, a correct digest does not prove the source had permission to supply that data. Evaluate identity, integrity, policy approval and authorization separately. MLflow dataset tracking records dataset name, digest, schema, source and run inputs to support lineage. Log a dataset with its training context and pin the immutable source revision. MLflow tracking is not itself an admission or approval enforcement service , and some dataset types use metadata only digests: verify the chosen digest's semantics before relying on it for integrity. Fallback policy: don't secretly use older data If no eligible snapshot exists: Default: Block retraining and continue serving the last validated model. Retry: Re run the admission check after the source is corrected or the required approval arrives. Prior approved version: Permit only if a documented exception authorizes a longer age or explicit fallback snapshot, with review of completeness and distribution risks. Emergency research use: Optionally allow a clearly labeled nonproduction diagnostic run , but do not treat its output as production approved or bypass access/data use constraints. Rebuild or backfill: Produce a new version from authorized data, rerun quality and approvals, then request admission normally. For example, using a two day old training snapshot might be acceptable for a low volatility monthly model but dangerous for a model driven by recent promotions or weather patterns. Even when an older version is approved, it should not silently bypass an age policy. Auditability and incident investigation Record: Requested dataset and logical training time. Candidate versions considered and why each was accepted/rejected. Applied admission policy revision and age thresholds. Actual quality report, approval evidence and authorization result. Immutable snapshot/digest bound to the data loader. Training run ID, model artifact and downstream registry relationship. Any override approver, expiration, justification and nonproduction constraints. Alerts for repeated rejection, expired approvals, unknown snapshots and failed integrity checks. Periodically sample finished training runs and verify their consumed dataset receipts still resolve to valid lineage and the approved snapshot identifiers. Detect mismatches between approved and actually read input versions. Trade offs and common mistakes Strict freshness: Protects relevance but may delay retraining when data updates are slow. Longer freshness windows: Improve availability but can degrade model relevance. Snapshot immutability: Improves reproducibility and audit but may increase storage and version management costs. Manual approval: Supports oversight for sensitive datasets but can become a bottleneck; automate routine checks with accountable policy. Overly broad allowlists: Can admit the wrong dataset; overly narrow allowlists may block legitimate evolution. Approval by filename: Renaming an object to approved.csv provides no trustworthy evidence. Using latest after approval: Causes time of check/time of use races. Rechecking only pipeline status: A completed ingestion job does not imply its dataset was approved for this training purpose. Ignoring revocation: A snapshot approved last week may have since lost permission for new training use. Trusting checksum alone: Matching bytes does not establish data quality or usage authorization. Treating model registration approval as data approval: These are different gates at different stages. In short: Admit training data only through an immutable snapshot receipt with verified freshness, quality, approval, lineage and access scope . Enforce the decision in the orchestrator and the data loader, block unknown or revoked versions, and audit the actual bytes and identity consumed by every model training run.

How do you coordinate multiple datasets with different update schedules?

Coordinate datasets with different update schedules by assigning each training run one logical cutoff time, then resolving the exact completed version of every required input that is valid for that cutoff. A scheduler should not simply join the newest hourly, daily, and weekly tables: those versions may refer to incompatible points in time and accidentally introduce future information. Question 111 covered whether a particular snapshot is approved for training, and Question 102 covered detection of missing or delayed arrivals. This lesson focuses on time alignment across multiple independently updated sources , release barriers, cross source consistency and targeted reprocessing. Beginner friendly example: delivery ETA model Suppose a food delivery service retrains an ETA model every morning from three upstream sources: Hourly traffic: Completed hourly traffic aggregates, normally finalized 10 minutes after each hourly window. Daily orders: Prior day order/label data, finalized at 06:00 UTC the following morning. Weekly restaurant profiles: A weekly snapshot published every Monday at 04:00 UTC, representing approved profile values effective from that Monday. The downstream training DAG starts its readiness check at 07:00 UTC on Tuesday, November 10, 2026 . It processes order events from Monday, November 9, 00:00–Tuesday, November 10, 00:00 UTC , using half open intervals [start, end). For this training slice it should use: 24 hourly traffic partitions covering 2026 11 09 00:00 through 23:00, each committed, with its own hourly interval. One complete daily orders partition for 2026 11 09. The Monday weekly profile revision effective for 2026 11 09, provided it was genuinely available at the corresponding historical prediction time. If the weekly dataset changes during the day, use effective time versioning rather than applying future changes to earlier events. It should not blindly choose the newest traffic partition at 07:00 on Tuesday: that would include Tuesday morning traffic after the Monday training examples and may leak future data into historical features. Numerical readiness example At the Tuesday 07:00 readiness check: Hourly traffic has 23 of 24 required committed partitions: 23/24 = 95.83% interval coverage. Daily orders has 1 of 1 required partition: 100% . Weekly profiles has 1 of 1 required effective snapshot: 100% . If the full 24 hour traffic coverage is mandatory, training must wait despite the other two sources being complete. Suppose the missing hourly traffic partition becomes committed at 07:18 . The readiness barrier then passes, subject to all snapshot integrity and approval checks. If company policy instead allows at most one missing traffic hour with a documented fallback , the workflow could proceed only after tagging that hour, imputing the approved fallback and evaluating bias from missing coverage. 95.83% time interval coverage is not necessarily 95.83% record, traffic volume or user coverage. For a weekly dataset covering 7 days , a daily run can reuse one weekly snapshot for each day within its validity interval; it should not force the weekly source to publish daily. The key concepts: five clocks and IDs Event time: When a delivery or traffic observation actually happened. Processing time: When the event was processed by a worker. Publication/commit time: When an entire source version became available and verified. Effective time: The business period during which a weekly restaurant profile or category mapping applies. Logical run interval: The historical period a training job intends to process, independent of when it actually executes. Keep logical interval start, logical interval end, snapshot id, valid from, valid to, committed at and source version identifiers in a consistent, documented timezone (often UTC). Do not confuse a weekly file's upload timestamp with the week its values describe. Airflow's data interval is the interval being processed; the run's logical date normally identifies its start under data interval timetables. Manual and trigger based runs may use different timetable semantics, so bind your own intended training cutoff explicitly when necessary. Build a cross source alignment contract For every dependency, specify: Hourly traffic Required interval: every hour intersecting the training window. Completion rule: committed hourly snapshots with applicable allowed lateness. Join rule: match traffic observations that were available at the original prediction time, not future derived hourly summaries. Gap policy: require 24/24 hours or use a reviewed fallback for sparse hours. Daily orders Required interval: the entire relevant business day. Completion rule: published daily snapshot with its validated coverage boundary. Join rule: join by order/entity IDs using the correct event time. Gap policy: normally treat a missing daily label partition as a mandatory blocker. Weekly restaurant profile Required interval: the appropriate business effective week or historical version. Completion rule: immutable revision with defined valid from and valid to. Join rule: point in time lookup of the profile effective and available when the order was placed. Gap policy: use a previous effective version only if explicitly allowed; never substitute a future revision retroactively. An important distinction: a snapshot can exist today but still be invalid for a historical training row if its features were not known at that row's prediction time. Coordinate both publication history and event/effective time semantics. Dependency barrier and consistent input manifest Create one immutable multi input manifest before training, such as: The IDs are illustrative. In production, resolve immutable snapshot/object version references, coverage evidence, digests and quality/approval receipts. The manifest as a whole is the reproducible input to a downstream run. Do not allow traffic revisions to change halfway through fitting. Simple Python multi source readiness check This is an educational set completeness gate , not a secure, production ready implementation. Real code must check each source's committed state, immutable IDs, quality/approval evidence, timestamps, row level point in time correctness and weekly effective window semantics. Also handle daylight saving changes if local business days—not UTC days—define the partitions; local days can contain 23 or 25 hourly intervals. Watermarks, lateness and cutoff times Streaming systems use an event time watermark to estimate progress through their input events. In Apache Beam, a watermark moving beyond a window boundary can trigger a result, but late events may still appear according to allowed lateness and trigger policies. A watermark is not absolute evidence that all source events exist. For hourly traffic: Hour window: [14:00, 15:00). Event time watermark reaches 15:05 , past the window end. The pipeline may materialize an initial completed window based on its trigger policy. Its agreed late data horizon might extend to 15:30 ; corrections received before that cutoff could require a revised hourly snapshot. The team must choose whether daily training at 07:00 consumes finalized hourly windows or provisional windows with revision tracking. If a provisional snapshot is allowed, save its exact revision and correction policy; never silently swap it later. Use separate cutoff time policies: When is a source considered sufficiently complete for a daily training run? How long should the orchestrator wait for corrections? At what point is a late correction assigned to a controlled backfill instead of modifying the current run? Prefer a business aligned cutoff that reflects expected labels and feature availability, not merely the first moment every API responds. Scheduling and orchestration An orchestrator such as Airflow can run hourly producer DAGs, daily orders jobs and weekly profile updates under independent timetables. A downstream training DAG should depend on specific logical data intervals , not simply on whether the latest producer DAG succeeded. A robust approach: 1. Publish completed source versions and partition identities to a trusted catalog/event source. 2. At each daily training interval, compute the exact hourly/daily/weekly dependencies. 3. Wait for the dependency manifest using sensors, dataset/asset events or a catalog query; apply a bounded timeout. 4. Validate the aligned version set, then publish the immutable multi input manifest. 5. Pass that manifest ID to training, rather than repeatedly reading source aliases like latest. 6. Monitor dependency readiness and produce alerts for a stalled run. The missing arrival incident mechanics were covered in Question 102. Airflow's timetable/data interval semantics determine which period a run processes. For unusually complex calendars, use a timetable or explicit logical interval mapping. Confirm the Airflow version's behavior before relying on a particular trigger/logical date association. Partial availability and graceful degradation Not all inputs deserve equal treatment: Mandatory labels or entity mapping: Block if incomplete, because the training target or join becomes unreliable. Optional external context: Permit a documented fallback if coverage and model impact are assessed. Weekly reference data: Reuse the latest historically valid prior effective revision when the source contract permits it. Provisional hourly signals: Use only if the training methodology explicitly supports snapshots that can later be corrected. Record source coverage alongside trained model metrics. A seemingly good score may hide excluded orders from missing traffic hours. In the example, skipping one missing hour in a peak period could eliminate disproportionately many delivery orders. The simple 23/24 coverage fraction cannot establish representativeness. Backfills across mixed cadences Suppose traffic for November 9 at 23:00 is corrected after a daily ETA training run has already been completed. Publish a new immutable traffic partition revision. Identify which daily training/feature windows depended on that hour. Determine whether the new revision changes model features materially or violates a correction policy. Trigger a targeted backfill only for affected daily intervals and downstream artifacts. Pin the revised traffic partition together with the same intended historical daily and weekly snapshot versions, unless those versions were also explicitly revised. Use an idempotency key such as eta:2026 11 09:manifest v2. Keep the original run's manifest and record why its successor was created. If a weekly reference dataset is corrected retroactively, a single revision can affect up to seven daily training intervals (or more when its effective period is different). Do not rerun all historical data without calculating the actual impacted dependency intervals. Be careful with latest inputs during replays: a backfill started months later must reproduce the original logical window, not leak current profiles or future labels into the past. Trade offs Wait for full cross source consistency: More reliable but delays training when one source lags. Use provisional windows: Faster updates, but increases revisions and potential backfills. Reuse prior weekly data: Stabilizes scheduling, but may miss real profile changes. Freeze snapshots: Reproducible experiments, with extra metadata/version storage costs. Recompute every affected day: Highest historical consistency, potentially expensive. Selective backfill: Saves compute, but requires trustworthy dependency lineage. Use one global cutoff for all sources: Simpler to explain, though sometimes overly strict for sources with different valid time semantics. Mistakes to avoid Joining latest hourly traffic to yesterday's training labels. Assuming three individually ready datasets are automatically temporally compatible . Treating a streaming watermark as proof of perfect completeness. Ignoring effective dates for weekly reference tables. Assuming every business day always contains exactly 24 local hours. Using current restaurant profiles for historical orders when those profiles changed. Overwriting previously selected source revisions during a running training job. Skipping the cross source manifest and losing the exact inputs needed to reproduce a model. Triggering indiscriminate backfills when only one partition changed. Treating a partially available high volume hour as harmless because 23 of 24 hour partitions passed. In short: Give each training run a logical interval, align hourly, daily and weekly sources by event/effective time, wait at a defined multi input readiness barrier, and freeze one consistent manifest of immutable source versions. Reconcile late corrections through targeted, auditable backfills rather than silently changing historical inputs.

How do you capture provenance for models trained using external datasets?

Provenance is a verifiable record of where training data came from, who supplied it, which exact version was used, how it was transformed, and which model was produced from it. For external datasets, it should also record the license and contractual permissions, because being able to download data does not necessarily grant permission to train, redistribute, or serve a commercial model with it. This lesson focuses on traceable origins and model to source relationships . Question 111 covered whether a dataset is approved to enter training, and Question 112 covered coordinating multiple update schedules. Provenance answers a different question: Can we explain and audit the exact external inputs that influenced this model months or years later? Beginner friendly example: external weather data for delivery ETAs A delivery company predicts arrival times using its own order records plus an external historical weather dataset. The pipeline receives: Internal orders: 80,000 labeled deliveries from company owned snapshot orders v42. External weather: 20,000 hourly weather observations from a third party provider. Training result: ETA model eta v18 produced by training run run 20261106 018. The company should save: Provider identity: Legal provider/operator name, verified service, dataset product ID and accountable internal owner. Permission evidence: Exact license/contract revision, allowed commercial training purpose, geographic scope, expiry, redistribution limits and attribution obligations. Source identity: Provider URL or dataset repository, exact release or commit ID, retrieval timestamp and object versions. Integrity: SHA 256 digest of the exact archived source bytes and a manifest of every file. Transform lineage: Versioned join, filtering and aggregation steps that combined weather with order data. Quality evidence: Validation run ID, outcome, row counts, expected coverage and relevant limitations. Training linkage: Immutable processed dataset ID, split IDs, training run, model artifact digest and registered model version. A mutable URL such as https://provider.example/weather/latest.csv is not sufficient provenance : tomorrow it may deliver different contents under the same name. Numerical example: reconcile source to training lineage Suppose the weather supplier provides 20,000 rows in an archived snapshot: 20,000 rows downloaded and their byte level checksum recorded. 800 duplicate source rows removed under an agreed deduplication policy. 200 rows rejected because of missing mandatory event timestamps. 19,000 rows remain in the cleaned weather dataset. Arithmetic: 20,000 800 200 = 19,000. The source to cleaning retention is 19,000 / 20,000 × 100 = 95%; 5% was removed by these transformations. The denominator is the weather dataset, not the 80,000 delivery orders. These counts are illustrative and assume the 800 duplicates and 200 invalid time rows do not overlap. If checks overlap, reconcile mutually exclusive reason counts. After a point in time join, suppose 76,000 of 80,000 orders obtain a valid weather feature: Join coverage = 76,000 / 80,000 = 95%. Unmatched orders = 4,000 , or 5% . The pipeline should preserve the join policy for those 4,000 orders: rejected, filled with a defined missingness indicator or imputed under an approved procedure. The final processed training snapshot gets its own checksum because cleaning and joining create new content. The original source checksum cannot prove that the final training table was built correctly. The minimum provenance chain In W3C PROV terminology, entities include dataset snapshots and models, activities include downloading and transforming data, and agents include the source provider and pipeline operator. Record what each activity used and what it generated . OpenLineage supports dataset versions, input/output datasets, job/run information and facets for quality measurements. <See the linked official references. Record the exact external source, not merely its homepage For each external dataset capture: Provider legal name, dataset title, product ID and versioned citation. Distribution URL, original filename or object store key and the retrieval method. Commit hash, immutable release, object version or vendor snapshot identifier . UTC retrieval timestamp and, where relevant, the provider's data coverage interval. Source document/version, API request parameters and source pagination identifiers, if reconstruction requires them. Total bytes, file count and per object checksums. Original schema and a documented collection description or dataset card. For datasets hosted on Hugging Face, use a pinned commit revision rather than main when you require an identical source revision. A commit reference identifies a repository state, while an independently verified byte digest protects the exact downloaded content. The provider can still later remove or restrict access, so choose an archive policy consistent with the license. Licensing and permissions are part of provenance Capture evidence , not just an unverified label such as license = open: Exact SPDX expression when applicable, plus a pointer or archived copy of the governing license text/version. Contract or data processing agreement ID for commercial feeds. Whether training, derivative artifacts, redistribution, model deployment and cross border processing are permitted. Attribution, citation, audit, usage reporting or deletion requirements. Personal data restrictions and consent/legal basis requirements where relevant. License expiration, upstream revocation mechanism and an internal accountable approver. Caution: A license covering the dataset does not automatically settle all rights in personal data, third party contributions, or model outputs. Seek appropriate legal review for ambiguous cases. Avoid publishing provider credentials, private contracts or protected source rows in public experiment metadata; store restricted evidence behind access controls and reference it by an internal document ID. Example versioned provenance manifest This is a generic design example , not a formal W3C/OpenLineage YAML specification. The placeholders are intentionally not valid integrity hashes. Real manifests must be bound to trusted permissions, snapshots and metadata records. Capture file checksums correctly A simple integrity helper for files received from a provider: Save the digest with the original archived file identity , its size, provider release and retrieval receipt. Verify it again before training or reprocessing. For a dataset with multiple files, record individual hashes and an ordered/canonical manifest digest; simply concatenating filenames from an unordered listing can produce unstable results. A matching digest proves byte level identity given a trusted expected digest . It does not prove provider authenticity, legal permission, data quality, or that transformations were correct. Verify source authenticity independently and protect provenance metadata against unauthorized edits. Connect provenance to MLflow training runs MLflow supports tracking datasets with name, source, digest, schema and profile , and linking them to training runs through mlflow.log input(). <See MLflow dataset tracking documentation. For a DataFrame that has already been loaded from a pinned, authorized source: The DataFrame df weather is illustrative and must already exist. This snippet logs metadata; it does not download, checksum verify, approve, or archive the source. Also, an MLflow dataset digest is an implementation defined dataset fingerprint and should not be assumed equivalent to a cryptographic SHA 256 over archived source bytes . Store an independent source file checksum. Log the final joined training dataset as a separate input when applicable, plus its source lineage and split IDs. The external weather input alone does not describe the complete training population. OpenLineage for transformation provenance A pipeline can emit OpenLineage job/run events describing: Ingestion job input : provider snapshot. Ingestion job output : raw archived dataset. Cleaning job input/output : raw to cleaned dataset. Feature building job inputs : cleaned weather and internal orders. Feature building job output : immutable joined training snapshot. Training run input : training snapshot; output : model artifact/metadata. Use standard dataset version and dataQualityMetrics/dataQualityAssertions facets where supported. Include job/source code versions and ownership. Lineage events allow an impact query such as: Which model versions depend on external weather release r3? Tools differ in which facets they capture automatically, so verify actual end to end coverage and fill gaps explicitly. Auditing, change tracking and source revocation Suppose an external provider announces that release weather 2026 10 r3 contains corrupted temperature readings or the license is withdrawn. A useful provenance system can: 1. Locate the provider and affected exact release ID . 2. Find raw/cleaned datasets derived from it. 3. Traverse transformation links to all dependent training snapshots. 4. Identify training runs and registered/deployed models that consumed those snapshots. 5. Determine whether the issue affects use permissions, integrity, accuracy, or all three. 6. Open an incident and pause new use of the affected dataset versions. 7. Follow contractual/legal direction for retention, deletion or retraining and document the decision. 8. Rebuild from an authorized corrected release when appropriate, then re evaluate downstream models. For example, if release r3 contributed to 4 training snapshots , each used by 3 experiment runs , the maximum direct run impact is 4 × 3 = 12 runs under the assumption of distinct run relationships; actual affected models must be deduplicated because one model may have multiple linked experiments. Track exact reverse edges instead of guessing that every current model used the latest external data. Revocation does not always mean every previously trained model must be deleted automatically; duties vary with licensing, privacy, contract and technical circumstances. Make that determination through the appropriate review process rather than guessing from a dataset tag. Reproducibility and retention trade offs Only keep external URL: Cheap, but weak; the link can change, disappear or return a new revision. Archive complete permitted source bytes: Best for later byte for byte checks, but requires storage, access controls and permission to retain copies. Store hash + manifest without bytes: Offers identity and traceability, but cannot recreate the dataset if the provider removes it. Track transformations and split IDs: Improves model reproducibility but adds metadata to maintain. Record every column level dependency: Valuable for impact analysis, but can be expensive; prioritize material features and sensitive sources. Use cryptographic signatures/attestations: Stronger tamper evidence when implemented correctly, but not a substitute for trusted provider identity or rights review. Coordinate provenance retention with artifact retention: if a production model remains in service, preserve the legally permitted provenance evidence needed to explain and govern it. Do not retain external data longer than applicable permissions allow simply because it would be convenient for reproducibility. Common mistakes Logging weather/latest.csv without a source release or immutable snapshot. Saving only an MLflow dataset digest and assuming it proves original file integrity. Tracking data provenance but omitting licensing/usage permissions. Preserving only the raw source while losing the transformation code and split definition. Treating a correct hash as evidence that the dataset is ethically or legally usable. Assuming a public download URL automatically grants commercial training rights. Logging protected license documents or confidential data into broadly visible experiment metadata. Copying current data into a historical run while labeling it with an older release ID. Failing to link derived datasets to model versions, making incident impact analysis impossible. Automatically discarding original run lineage when an upstream dataset is revoked. In short: A good external data provenance record binds provider, rights, exact source revision, original bytes, transformation history, quality evidence, final training snapshot and model run . Capture it at ingestion and every important transformation boundary so audits, reproducibility and incident investigations can follow the full chain.

How do you design a complete automated experiment-to-registry workflow?

An automated experiment to registry workflow is a reproducible, orchestrated chain that turns an approved dataset snapshot into a trained, evaluated, traceable model version in a registry. The critical design principle is to pass immutable identities and verified evidence between stages rather than mutable paths, informal approvals or unchecked files. This question brings together the building blocks from earlier lessons. Question 110 examined the individual evaluation gate and Question 113 examined external data provenance . Here we focus on how the entire system is composed, what each component hands to the next, and how the workflow recovers from failures without registering an unapproved model. An end to end reference workflow Registry registration is not production deployment. A model can be approved as a candidate in a development/staging registry while deployment remains a separately controlled process. Some organizations retain rejected versions in an experimental registry; the rule is that an unapproved candidate never becomes production eligible through this workflow. Step 1: Trigger a reproducible pipeline run Common triggers include a completed upstream data snapshot, a scheduled retraining date, an approved code change or an explicit human request. Create a run request containing: Unique pipeline run ID and idempotency key. Logical data interval, business purpose and environment. Required dataset identifier and allowed approval policy. Training code commit, container image digest and configuration version. Baseline model version and evaluation policy revision. Accountable owner and requested action: experiment, approved registration or diagnostic run. An orchestrator such as Kubeflow Pipelines , Airflow, Dagster or Prefect schedules dependencies, retries and resource allocation. The orchestrator should not be the only place where the dataset and model release policies are enforced. Step 2: Resolve approved input snapshots Run the training admission check from Question 111, then return an immutable snapshot reference , a content identity and an evidence receipt. Example: Dataset: delivery orders. Snapshot: orders 20261105 v3. Approval evidence: approval 450. Data quality report: dq 981. Code revision: git a1b2c3. Do not pass orders/latest to the training step after the gate has checked snapshot v3; it could silently change between tasks. The trainer must receive the same exact version the gate approved. Step 3: Prepare features and persistent splits A preprocessing component receives the approved snapshot ID and emits: A versioned feature dataset, with schema and transform code revision. Training/validation/holdout split IDs. Row counts, coverage summaries and leakage checks. Immutable artifact locations and digests. For time dependent tasks, define chronological and entity aware splits to avoid leakage. Feature and split artifacts must be reusable across retries: two executions of the same logical run should not randomly create unrelated training cohorts. The preprocessing stage should be idempotent , publishing only when its complete output and metadata have been validated. Step 4: Train and track the experiment The training component reads the pinned features and produces: Trained model artifact and model signature. Run parameters, fixed seeds and environment metadata. Training metrics, logs and any required checkpoints. Unique MLflow run ID linked to input datasets and experiment. Artifact digest and location in durable storage. MLflow distinguishes its tracking backend (run metadata, parameters, metrics) from the artifact store (model files, plots and other larger outputs). Configure both for durable access before automated jobs start; worker local storage is not adequate when pods can disappear. A simple code sketch: The training function and upstream snapshot resolution are placeholders. This sketch logs a model but intentionally does not register it yet ; the policy gate should run before approval for registration. The name parameter reflects modern MLflow model logging APIs; exact integration can differ by installed MLflow release. Step 5: Evaluate candidate versus the approved baseline Evaluate the candidate and currently approved baseline on the same immutable holdout with identical metric definitions. For a delivery ETA model, a sample evidence package could include: Overall validation MAE and final holdout MAE. Baseline comparison on matched rows. Peak hour and rural delivery slices. Relevant prediction latency and model loadability results. Evidence that features did not contain unavailable future information. Confidence/uncertainty estimates where the release policy requires them. The model evaluation procedure and threshold selection belong to Question 110; here the key is that the evaluator publishes a signed or trusted evaluation receipt bound to run id, model digest, holdout snapshot and gate policy version. Registration must consume that receipt rather than trust a plain passed=True string supplied by arbitrary training code. Numerical example: from 4 candidates to 1 registry version Assume an ETA training pipeline produces four candidates against the same holdout dataset of 5,000 deliveries . The approved baseline MAE is 5.0 minutes . The example policy requires overall MAE ≤ 4.7 minutes and a rural slice MAE ≤ 6.0 minutes . Candidate A: overall 4.9 , rural 5.5 → overall gate fails. Candidate B: overall 4.6 , rural 6.4 → slice gate fails. Candidate C: overall 4.5 , rural 5.8 → both pass. Candidate D: overall 4.7 , rural 5.9 → both pass, assuming an inclusive threshold. If the predeclared selection rule chooses the eligible candidate with lowest overall MAE, candidate C wins. Its absolute improvement is 5.0 4.5 = 0.5 minutes, or 0.5/5.0 × 100 = 10%. Only one winner is selected for approved registration by this policy, although the experiment tracker retains all four candidate run records. This example does not establish statistical significance or safety; those checks must be included if required by the release contract. Step 6: Register the approved candidate as a version After the approval receipt succeeds: 1. Confirm model artifact digest, model signature and loading compatibility. 2. Check whether an approved registry version for the same run/model digest/approval already exists. 3. Register a logged model under an owned registered model name, e.g. delivery eta. 4. Record the returned registry version number and link it to run ID, model identity, snapshot ID and evidence receipts. 5. Apply controlled tags such as validation status=approved and approval record id=.... 6. Make it available to a separate staging/deployment process. For modern MLflow APIs, a logged model can be registered by URI: The logged model uri is the URI returned by log model from the approved training run. Registration is a side effect and should be guarded against duplicate retries. In a registry supporting version tags and aliases, assign them through controlled policy aware code. An alias such as champion is a mutable pointer, not an immutable audit identity; an approval workflow should record the actual version number. MLflow's Model Registry supports model lineage, versions, metadata tags and aliases. Its current workflows describe registering logged models and organizing promotion through controlled environments. <See MLflow Model Registry references. What each stage hands to the next Make all outputs structured and versioned: Trigger → admission: A run request with logical time, code/config revision and identity. Admission → preprocessing: Approved immutable snapshot, data quality receipt and source identity. Preprocessing → training: Feature snapshot, split IDs and schema/validation receipt. Training → evaluation: Logged model URI, run ID, artifact digest and training evidence. Evaluation → registration: Gate result, model digest, holdout ID, metric package and approval signature/record. Registration → deployment workflow: Registered model name, immutable version number, model signature and evidence pointers. Keep these contracts small and store large files in durable artifact storage. A scheduler should pass references rather than huge DataFrames or model blobs between task nodes. Failure handling without inconsistent registry state Input admission fails: Do not train; mark run BLOCKED INPUT with reason. Preprocessing fails: Retry transient I/O or worker errors according to policy; avoid reusing partial feature output. Training fails: Preserve useful diagnostic evidence and checkpoints; do not register an incomplete model. Evaluation fails: Preserve the candidate in experiment history, mark it REJECTED or HOLD, and emit no approval receipt. Registry API times out: The registration may already have succeeded. Reconcile by run/model identity before retrying; otherwise duplicate registry versions could be created. Workflow coordinator crashes: Reconstruct committed task states and artifact references from durable metadata. Resume from the last verified stage instead of rerunning every successful step blindly. Notifications fail after registration: Retry the notification independently without creating a second model version. A good approach is to maintain a durable state machine , for example: REQUESTED → INPUT APPROVED → FEATURES READY → TRAINED → EVALUATED → APPROVED → REGISTERED with explicit alternate outcomes FAILED, REJECTED and CANCELLED. Prevent invalid transitions such as TRAINED → REGISTERED without required evaluation/approval receipts. Auditability and reproducibility For each registered version, a reviewer should be able to answer: Which run and source code/container versions created it? Which exact dataset snapshot and preprocessing/split versions were consumed? Which parameters and RNG seeds were used? Which model artifact was registered and what is its digest? Which baseline and holdout were used? What were the observed metrics, slice results and gate decisions? Who or what approved registration, using which policy version? Were there retries, exceptions or failed intermediate attempts? Has the model been deployed, and where is the last approved rollback version? Store identifiers and immutable evidence rather than duplicate large datasets everywhere. Log run to model lineage in the registry and preserve the supporting evidence under the retention policy from Question 109. Monitoring and operating the workflow Monitor: Trigger to registration duration and stage by stage time. Number of successful, failed, blocked and rejected runs. Dataset admission and preprocessing failures. Model evaluation pass/fail and policy version distributions. Artifact upload integrity, missing URIs and model loadability. Registry registration latency, duplicate registration attempts and permission errors. Orphan candidate artifacts and runs without completed decisions. Cost per successfully approved model and resource utilization. Drift or evaluation regressions that should trigger a new training cycle, without automatically approving deployment. Alert on repeated failures or broken data/model integrity, and correlate multiple downstream errors with the same run ID. The process should be observable from the initial trigger to a verified registry object, not merely until the training container exits. Practical technology choices and trade offs Airflow/Prefect/Dagster: Useful for orchestration, dependencies, scheduling and recovery; they do not replace a model registry. Kubeflow Pipelines: Provides component based pipeline DAGs and ML artifact tracking, which can integrate with external tracking/registry tools. MLflow: Tracks experiments and model artifacts and provides a registry; it does not by itself orchestrate every external data or approval dependency. Manual review: Valuable for high risk candidate approvals, but can slow routine updates. Fully automated approval: Faster and consistent for well defined low risk policies, but requires strong data, metric and policy versioning. One giant training task: Simple at first, but weak failure diagnosis and recovery. Many tiny tasks: Easier caching and retries, but more operational overhead and more contracts to maintain. Start with a modest number of well defined stages and explicit receipts; refine granularity only where it improves reliability, cost or ownership. Mistakes to avoid Training from latest data after approval applied to a different snapshot. Using a training metric as a substitute for holdout evaluation. Automatically registering a model while training is still writing artifacts. Assuming registering a version means it is safe to serve production traffic. Retrying a timed out registry call without checking whether it already succeeded. Keeping only the winning model and losing evidence of rejected trials. Changing evaluation thresholds mid run without a new policy revision. Giving every task registry write credentials instead of restricting them to the approval/registration stage. Reporting the pipeline green when the registry version or evidence receipt is missing. Passing a mutable alias as the only record of what was approved. In short: Treat experiment to registry as an evidence driven state machine . Trigger a reproducible run, pin approved inputs, transform and train with lineage, log durable artifacts, evaluate against a fixed policy, and create an auditable model registry version only after all required approval evidence passes . Keep production deployment as a separate controlled decision.

How do you decide whether a model should run as a batch job or online service?

Choose batch inference when predictions can be computed ahead of time for many records; choose online inference when a client needs a response during a live request. The decision depends on response time SLA, traffic shape, input freshness, throughput, cost and failure handling—not the algorithm alone. This begins Model Deployment & CI/CD Automation . Question 114 connected experiments to the model registry. This lesson assumes an approved model exists and asks how predictions should be delivered . Beginner friendly example: delivery ETA A food delivery company uses the same model for two purposes: Live checkout: A customer requests a delivery estimate using current driver location and traffic. The service aims for p95 end to end response latency below 200 milliseconds . Choose online inference . Next day courier planning: Operations wants predicted durations for 1,000,000 hypothetical deliveries by 07:00 the following day . Choose scheduled batch inference . Batch and online predictions might share model weights but have different service contracts. Batch inference: scheduled scoring Typical flow: Start hourly, nightly, weekly or when a committed input snapshot appears. Process large datasets in partitions; parallelize where supported. Write predictions with stable entity IDs, input snapshot, model version and scoring time. Publish outputs only after required partitions finish and validation succeeds. Measure job completion SLA , output count, backlog and cost per record. Stop compute after scoring when infrastructure permits. Best for: Precomputed risk scores, nightly recommendations, campaign targeting, planning dashboards and bulk document classification. Batch is not intrinsically slow per record; it optimizes large scale processing rather than interactive response latency. Online inference: serve an API Typical flow: Expose a secure endpoint such as POST /predict. Keep model ready in memory or use suitable managed/serverless infrastructure. Budget for network, authentication, preprocessing, feature retrieval, model execution and serialization. Scale replicas based on peak load, latency distributions and saturation. Enforce request deadlines, authorization, input validation, rate limits and controlled rollback. Measure p50/p95/p99 response latency, timeouts, RPS and error rate. Best for: Checkout ETA, live fraud decisions, instant recommendations and interactive personalization. Numerical example: throughput and latency Suppose checkout traffic peaks at 120 requests/second . A tested server replica handles 40 requests/second while meeting the required latency under representative conditions. Absolute theoretical minimum: ceil(120 / 40) = 3 replicas. With 70% target steady utilization , planned effective capacity is 40 × 0.70 = 28 RPS each. Planning estimate: ceil(120 / 28) = 5 replicas. Five replicas are a starting capacity estimate , not proof of a 200 ms SLA. Test request bursts, concurrent feature reads, autoscaling lag and p95/p99 behavior. For the nighttime planning batch, suppose: Input: 1,000,000 records . Four workers, each scoring 5,000 records/minute . Ideal aggregate throughput: 4 × 5,000 = 20,000 records/minute. Ideal scoring time: 1,000,000 / 20,000 = 50 minutes. Startup, validation and output publication: 15 minutes . Estimated batch duration: 65 minutes . To finish by 07:00 , the theoretical latest start is 05:55 , without any margin for retries, variability or input delays. A real production schedule should include a buffer. Different SLAs: The online service must satisfy per request response time ; the batch job must finish its whole dataset before a deadline. Estimate infrastructure costs The following rates are illustrative assumptions , not current cloud prices. Online: Two always on instances at $0.40 per instance hour , in a 30 day month: 2 × 24 × 30 × $0.40 = $576/month. Batch: Four workers at $0.40 per worker hour , active 65 minutes daily : 4 × (65/60) × 30 × $0.40 = $52/month. The batch estimate is cheaper for this particular nightly workload, but both exclude storage, network, feature lookups, replicas during peak hours, logs, startup and operational costs. Online serverless deployments can reduce idle expense but introduce cold start and resource constraints. Repeated huge batches can be more expensive than efficiently utilized always on serving. Compare total cost per successful prediction , deadline success , and latency compliance , rather than instance prices alone. Prediction freshness Online inference can include live signals, if feature lookup is current and fast enough . Batch results may remain in use until the next completed run. A nightly 01:00 job can leave predictions nearly 24 hours old before the next refresh, and older if the next job fails. A freshly scored output may still be stale if the input features were old. Record the feature observation time, model version and prediction generation time. Enforce consumer specific maximum permitted prediction age. Monitoring and reliability Batch monitoring: Completed versus expected input/output partitions. Records attempted, succeeded and failed. Queueing time, throughput, retry counts and completion time SLA. Output integrity and version lineage. Scheduled start/finish lag and cost per scored record. Online monitoring: RPS, concurrency, p50/p95/p99 latency and timeout rates. Server saturation, autoscaling events and availability. Feature freshness, model version and input/prediction drift. Correctness after delayed labels and serving costs. Batch recovery: Retry transient processing failures, resume eligible partitions with idempotency keys and publish only complete committed outputs. Prior outputs may be reused only when the consumer permits their age. Online recovery: Use bounded retries, timeouts, health checks, capacity controls, rate limiting and documented fallbacks. A response timeout does not necessarily prove no side effect occurred; keep non idempotent downstream actions safe. Hybrid and asynchronous alternatives A hybrid approach can precompute slowly changing restaurant baseline preparation times nightly , then combine them with live driver and traffic features during the online ETA request. This reduces online work while preserving request time personalization. Track version and freshness of both the precomputed and live inputs. Asynchronous inference is distinct from scheduled batch scoring: a caller submits a heavy or long running request, gets a request/job ID, and retrieves its output later. AWS SageMaker distinguishes real time , serverless , asynchronous and batch transform inference options. Async APIs are covered in Question 117; production batch pipeline integration is covered in Question 116. Decision checklist 1. Must the user wait for the prediction now? Strong signal for online. 2. Can the result be precomputed from stable inputs? Strong signal for batch. 3. What is the SLA? p95/p99 response latency for online; end to end job completion for batch. 4. How fast do inputs change? Define a maximum permissible prediction age. 5. How many requests or records? Estimate peak RPS, batch volume and concurrency. 6. What infrastructure is required? Include memory, CPU/GPU, model loading and feature retrieval. 7. What does full lifecycle cost look like? Include idle capacity and repeated jobs. 8. What happens on a missed deadline or outage? Specify fallback and retry policies. 9. Could a hybrid or asynchronous approach satisfy the requirements better? Common mistakes Paying for always on endpoints when scoring is only needed once daily. Assuming batch is suitable for a user facing response requiring immediate live inputs. Measuring only model execution time instead of complete API latency. Sizing online capacity from average RPS and ignoring traffic peaks. Publishing partial batch outputs without a completion marker. Calling a prediction fresh while its feature data is outdated. Assuming all serverless endpoints meet strict latency without cold starts. Using expired batch predictions silently instead of applying a freshness policy. Failing to record model and dataset versions for prediction audit. In short: Choose batch for precomputable, bulk predictions with a completion deadline; online for interactive, low latency inference; and hybrid/async when latency, workload and freshness requirements call for a combination.

How do batch prediction jobs fit into production data pipelines?

A production batch prediction job is a scheduled or event triggered stage in a data pipeline that reads a defined input snapshot, applies an approved model, and publishes a validated prediction dataset for downstream systems. Unlike an interactive API, it must manage partitions, dependency readiness, deadlines, retries, and atomic publication across the entire dataset. Question 115 explained when to choose batch versus online serving . This lesson explains how batch scoring actually operates as one dependable production data pipeline stage , without revisiting the architecture decision. Beginner friendly example: nightly delivery risk scores A delivery platform predicts whether deliveries scheduled for the following day are likely to be late. At 01:00 UTC it receives a complete, approved snapshot of tomorrow's expected orders. Its batch scoring pipeline must generate one risk score per eligible order, making the results available to an operations dashboard by 03:00 UTC . The components are: Input: 1,000,000 order records, split into 20 immutable partitions of 50,000 records each. Model: Approved registry version delivery delay risk/27, not a mutable latest alias. Preprocessor: Versioned feature transformation code and schema. Output: Partitioned prediction records with order ID, probability, model version, input snapshot, and scoring timestamp. Consumer: Operations dashboard and courier planning service that read only the committed output dataset. If the model is upgraded tomorrow, today's published predictions remain attributable to version 27. The old prediction output must not silently change just because an alias moves to version 28. End to end batch pipeline The pipeline should run with a logical date distinct from wall clock start time. A retry tomorrow for the November 6 scoring interval should target the same intended input/model version unless the operator explicitly requests a corrected revision. Scheduled versus event driven triggers Scheduled: Run nightly at 01:00; useful for a known business cutoff. The run still needs a readiness barrier, because wall clock time does not prove data exists. Event driven: Start after the upstream data producer publishes a committed snapshot event. Verify the event's snapshot identity and completeness instead of trusting a notification alone. Hybrid: A nightly deadline with an upstream completion event: run when input is ready, and alert or escalate if the deadline approaches. Backfill trigger: Re score a specific historical interval because data or the model input contract was corrected. Give it its own output revision. The scheduler coordinates execution. It does not replace data validation, model authorization or output transaction semantics. Numerical example: partition throughput and deadline Assume 1,000,000 orders , 20 partitions of 50,000 records , and 4 workers . Each worker processes a 50,000 record partition in 8 minutes , including feature transformation and inference. Each worker's throughput = 50,000 / 8 = 6,250 records/minute. Aggregate ideal throughput = 4 × 6,250 = 25,000 records/minute. Partition waves = 20 / 4 = 5. Ideal processing duration = 5 × 8 = 40 minutes. If readiness, startup, validation and final publication add 15 minutes , the ideal end to end time = 55 minutes . With a pipeline triggered at 01:00 UTC , the illustrative completion time is 01:55 , leaving 65 minutes before the 03:00 SLA deadline . Real executions may take longer because of skewed partitions, I/O contention, worker loss, straggling tasks and queue delay. Monitor p95 job duration and deadline misses , not only ideal throughput. Now suppose partition 17 fails once after 6 minutes and needs its full 8 minute attempt again. If capacity is available, the retry can extend makespan; the exact extension depends on the worker schedule. It is incorrect to assume every retry always adds exactly 8 minutes to the whole run. Upstream contracts and input validation A batch model must know what it is scoring: Logical interval and snapshot ID: The source must be complete for the intended scoring date. Stable entity IDs: Every output must map back to exactly the correct input record. Schema: Validate names, types, units, nullable columns, categorical values and feature ranges. Coverage: Compare expected versus actual partitions and record counts. Feature availability: Ensure the preprocessor uses the same permitted feature definitions as the model was built to accept. Data use rules: Enforce approved sources and permissions where applicable. Missing data policy: Fail, quarantine specific records or use approved imputations—never silently drop critical input rows. Example: the model expects distance km as a number. A new upstream file starts supplying distance meters with the same column name. A schema/type check alone might not catch the semantic change, so add unit and distribution checks. Model version selection and reproducibility Resolve the model version once at the start of the logical run , then persist: Registry name and immutable version . Model artifact URI and expected digest. Preprocessing signature, feature schema and container/code version. Input dataset version and transformation revision. Scoring configuration, output schema version and pipeline run ID. Avoid dynamically resolving champion or latest inside each partition worker. If the alias changes midway through a million record job, some partitions could be scored by different model versions, yielding an inconsistent prediction dataset. Partitioned scoring and record identity Partition by a stable business key or input file partition, then make the output independently retryable. Each output row should contain, where appropriate: order id or another durable row ID. prediction and optional confidence or probability. model version, input snapshot id, logical scoring date. scored at, run id and output schema version. Error/status field for intentionally quarantined records. Prefer one well defined result per input key for workloads that require exact mapping. Deduplication should use a business key plus the intended scoring window and output revision—not simply whichever write happened last. Azure ML batch endpoints process data in mini batches, can scale across worker nodes, and store outputs in a configured datastore. Their controls include mini batch size, retry settings and error thresholds. Production correctness still depends on the application defining the proper output reconciliation policy. Idempotency, retries and atomic publication Idempotency means repeating the same logical scoring job or partition does not duplicate externally visible predictions or change which version is considered final. A useful identity is: scoring date + input snapshot id + model version + scoring code version + output revision. For example: 2026 11 07:orders v51:delay risk 27:scorer v8:r1. A safe output strategy: 1. Assign each attempt an isolated staging path. 2. Retry only eligible transient errors such as intermittent object store or worker failures. 3. Write each partition's results and a completion receipt with row counts, checksums and model/input identities. 4. Validate that all expected partition receipts exist and are compatible. 5. Reject or quarantine invalid predictions according to the preapproved error policy. 6. Publish a small committed manifest or perform a transactional table commit. 7. Allow consumers to read only committed output revisions. An object store directory is not automatically atomic. For object storage, publish an immutable manifest/pointer only after every staged object has been verified; for a transactional table, use its atomic commit mechanism. Never expose partial scores just because the first worker succeeded. Simple Python illustration: partition accounting This is only a numerical illustration , not a production gate: counts can match even if there are duplicate or missing entity IDs. A real gate verifies partition identity, record level uniqueness, valid prediction ranges, object hashes, authorized versions, approved missing row thresholds and atomic visibility. Quality gates before consumers see outputs Validate: Every mandatory input partition has a corresponding valid output partition. The number of scored + explicitly quarantined records reconciles with inputs. No unexpected duplicate IDs or invalid probabilities (for classification, probabilities in [0,1]). Feature and model signatures match; no mixed model versions. Outputs have expected columns and valid partition metadata. Publish time remains within the business SLA. The expected output manifest can actually be read by the downstream consumer. A successful compute exit code does not prove the prediction dataset is complete. AWS SageMaker documentation notes that a batch transform job can leave outputs for successfully processed files while others fail. Treat provider job status and output validation as separate signals. Backfills and controlled re scoring Suppose a missing input partition for November 5 is corrected on November 7. Do not overwrite the original published prediction set invisibly. Identify the affected logical scoring interval. Create a new immutable input snapshot or correction revision. Choose the model version policy: original version for strict historical replay or a documented new version for deliberate re scoring. Run only affected partitions if the scoring contract permits partition replacement; otherwise rebuild the dataset consistently. Publish revision r2 , with a new manifest, lineage and explanation. Switch downstream readers to r2 only through a controlled promotion transaction. Preserve r1 as needed for investigation or audit. A replay must not accidentally score historical data with today's mutable latest feature dataset. Monitoring and SLAs Track data readiness latency , queue time, start time, total processing duration, partition throughput and end to end deadline success . Also monitor: Partition completion, retries and error/quarantine counts. Prediction distribution changes, null rates and unexpected category shifts. Model version drift or signature mismatches between workers. Input/output record reconciliation and duplicate counts. Storage/compute cost per million scored records. Backlog and last successfully published scoring interval. Downstream read failures, manifest corruption and delayed consumption. The important availability signal is not merely “job succeeded,” but “a valid complete prediction revision was published and can be consumed before its SLA.” Production trade offs Larger partitions: Fewer coordination tasks but bigger retry cost and risk of memory pressure. Smaller partitions: Better retries and load balance but more task and storage overhead. Strict all or nothing publication: Strong consumer consistency, but one failed partition delays everything. Partial publication: May be acceptable for specifically designed noncritical outputs, but requires explicit coverage metadata and consumer tolerance. Pinned model version: Consistent and auditable; a newly approved model takes effect on the next planned run, not midrun. Event driven launch: Reduces idle waiting but needs reliable event and version verification. Scheduled launch: Predictable business cadence but may wait when source data is late. Managed batch inference: Reduces infrastructure operations but still requires ownership of data contracts, schema compatibility and audit records. Common mistakes Letting every worker resolve a mutable model alias independently. Scheduling the job without verifying the required input snapshot is ready. Comparing only job exit status rather than record level output completeness. Writing retry results directly into one shared consumer visible path. Allowing duplicate predictions for the same entity and logical window. Publishing predictions as soon as some partitions finish. Losing the input snapshot ID or model version from output metadata. Re scoring historical windows with current features without a new declared revision. Treating a 03:00 worker finish as success if final output was not committed until after the SLA. Ignoring partial output artifacts after job failure. In short: Treat batch inference as a versioned, idempotent data product pipeline : wait for the right upstream snapshot, pin the model, validate and partition inputs, score reliably, reconcile every output, publish atomically, and monitor the committed dataset against a business deadline.

What is an asynchronous prediction API and when should it be used?

An asynchronous prediction API accepts a prediction request, returns a tracking ID quickly, and processes the actual ML inference later through a queue and worker. The caller can check job status or receive a notification when the result becomes available. It is useful when prediction takes longer than a normal HTTP request should remain open, payloads are large, traffic is bursty, or the caller does not need an immediate answer. This is request driven asynchronous inference , not synchronous online inference (Question 115) or scheduled bulk batch scoring (Question 116). One client request typically creates one job or one explicitly scoped job containing multiple items; the job is processed independently of the initial HTTP connection. Beginner friendly practical example A logistics company offers an API that analyzes a customer uploaded delivery route document and predicts which routes may be delayed. Each uploaded file requires document parsing, feature extraction and model inference and can take 45 seconds . Keeping an ordinary HTTP request open for 45 seconds risks browser, gateway or load balancer timeouts and ties up request serving resources. Instead: 1. The client submits the document or an authorized upload reference with a unique idempotency key. 2. The API validates authentication, authorization, size, file type and processing eligibility. 3. The service durably records a PENDING job and enqueues work. 4. It returns HTTP 202 Accepted , a job ID and status URL. 5. A worker claims the job, loads the approved model and processes the input. 6. The worker stores an output artifact and marks the job SUCCEEDED or FAILED . 7. The client polls status or receives an authenticated webhook, then obtains the result. The initial HTTP response says accepted, not completed . MDN documents that HTTP 202 is noncommittal: the later operation can still fail. Do not return a success style prediction body before inference actually finishes. API request and response contract A representative request: After validation and durable job acceptance , respond: The placeholder token and URI are illustrative. The Location reference and Retry After polling hint are useful API conventions, but 202 alone does not guarantee the server actually queued the task successfully. Use a transaction/outbox or reliable enqueue procedure so accepted jobs are not lost between recording the job and publishing to the queue. A status endpoint might return: When completed, the same endpoint returns SUCCEEDED with an authorized result reference , or a redirect to the result resource. A FAILED job must expose an appropriate safe error code; avoid exposing stack traces, signed storage credentials or sensitive model internals. Architecture and handoffs Important boundaries: API nodes should be stateless; job status must survive API restarts. The queue contains a job ID or trusted payload reference , not raw secrets or necessarily the entire uploaded file. Every job is bound to a tenant/owner and immutable model version , authorized input and execution policy. The worker stores result bytes durably before publishing SUCCEEDED. The client reads results using authorization independent of whether it knows the job ID. Store an output expiry and deletion policy; async results should not remain accessible forever by default. Numerical capacity and queueing example Assume a document inference worker takes 45 seconds per request , on average, and can process one at a time. One worker's ideal service capacity: 60 / 45 = 1.33 jobs/minute. With 6 workers : 6 × 60 / 45 = 8 jobs/minute. Suppose average arrivals are 6 requests/minute : utilization is approximately 6 / 8 = 75%. A sudden burst of 60 extra jobs arrives while all workers are busy. With the simplified assumption of no continuing arrivals, the burst requires about 60 / 8 = 7.5 minutes of aggregate worker service. The last burst jobs may wait much of this interval before starting. This is an idealized workload calculation , not a precise queue latency prediction. Real wait times depend on job duration variability, scheduling, worker startups, priority and continued arrivals. At arrival rates above 8 jobs/minute , the backlog grows unless capacity is increased or admission is restricted. For an example service level objective (SLO), choose: API acknowledgment: p95 under 500 ms for accepted requests. Job completion: 95% of eligible document jobs finish within 5 minutes , measured from acceptance. Result availability: Completed results retrievable within 10 seconds of final state commit. The example SLO can fail during the 60 job burst even though API acknowledgment remains fast. An async API does not eliminate processing latency—it removes it from the initial HTTP response. Be explicit about both acknowledgment latency and end to end completion time. Polling versus webhooks Polling : Caller periodically requests GET /prediction jobs/{id}. Easy for mobile/web clients, works across firewalls and supports reconnects. Can produce many empty status checks and increased API load. Use Retry After, exponential backoff, jitter and a maximum polling duration. Webhooks : Server sends an event to a registered callback when the job completes or fails. Convenient for server to server integrations; reduces polling. Callbacks can be delayed, delivered more than once or fail. Authenticate webhook signatures, validate destination ownership, restrict callback destinations to prevent SSRF, retry deliveries with backoff, and let consumers deduplicate notification IDs. Best practice: expose a durable status endpoint even when webhooks are enabled . A webhook is notification, not the sole source of truth. Microsoft’s async request reply pattern documents polling through a status resource after an HTTP 202 acknowledgment. Status state machine Persist status transitions with timestamps, attempt numbers and reasons. A worker must not overwrite CANCELLED with SUCCEEDED if the cancellation won the race; use conditional state transitions or fencing tokens. Terminal outcomes should not be silently reset, and retries should remain auditable. A result resource is available only for a SUCCEEDED job with verified durable output. A failed job is not equivalent to an HTTP 500 for the original POST, because that POST already returned 202. Idempotency and retries Suppose the client submits a job, the server accepts it, but the acknowledgment is lost. The client retries POST. Without idempotency, the system might process the document twice and charge twice. Accept an Idempotency Key scoped to tenant, endpoint and a defined retention period. Bind it to a canonical payload digest; if the same key is used with a different request, return a documented conflict rather than silently reusing an unrelated job. Return the same job ID for an equivalent retry. Use a transactional uniqueness constraint for key ownership, not an in memory dictionary. Treat queue delivery as at least once unless exactly once semantics are demonstrably provided end to end. Make worker claims lease based and writes idempotent; duplicate workers should not publish conflicting results. Separate queue message delivery attempt from logical prediction job ID and model inference attempt . For transient storage/network errors, retry with bounded exponential backoff and jitter. For invalid inputs, unsupported model versions, revoked permissions, or deterministic parsing errors, fail with a meaningful final error rather than retrying endlessly. Use a dead letter queue or failure store for jobs exhausted after the configured retry limit, while preserving authorization and privacy constraints. Timeouts, cancellation and backpressure Use separate limits: API deadline: Maximum time to validate and durably accept a job. Queue wait deadline: How long the job may remain pending. Execution timeout: Maximum worker run time. End to end job deadline: Maximum elapsed time from acceptance to final result. Result retention time: How long completed output can be downloaded. A client disconnect must not automatically cancel a durably accepted job unless that is an explicit contract. Support a cancellation endpoint if needed; cancellation may be best effort and must handle races with completion. When arrivals exceed capacity: Cap per tenant and global queued jobs. Apply quotas and fair scheduling; avoid one customer monopolizing GPU workers. Return 429 Too Many Requests or 503 Service Unavailable with a documented retry hint before acceptance when overloaded, rather than issuing a 202 for work the system cannot retain. Scale workers based on queue depth, oldest job age and service time. Reject oversized payloads and protect against retry storms. Merely adding workers does not always help when a shared model server, database or storage API is the real bottleneck. Example Python API contract (illustrative) This illustrates the endpoint, not executable production code : durable create or find job and authorization are intentionally unspecified, and a production response should also set a Location header. Never enqueue first and return 202 if a subsequent database failure would leave an untracked job. Job APIs must enforce tenant isolation for status/results and ownership of external input references. Security and result handling Authorize submission, polling, cancellation, callbacks and result retrieval independently. Use unguessable job IDs but never treat possession of an ID as sufficient authorization . Validate input URIs, prevent arbitrary server side URL fetches, and use short lived access where appropriate. Encrypt payloads and prediction artifacts, avoid secrets in queue messages, and limit who can invoke each model. Publish only validated outputs; persist a checksum, content type, ownership and expiry. Use tenant scoped storage paths or access control; expired signed URLs should be regenerated only after authorization. Apply deletion and data retention policies to input and output artifacts. For user provided documents, logs should record sizes, statuses and trace IDs without copying sensitive document contents. Monitoring and operations Track both fast acknowledgment and slow execution: POST accept latency, rejects, auth failures and idempotency conflicts. Queue depth, age of oldest queued job , enqueue to start waiting time. Active worker count, utilization, throughput and autoscaling delay. End to end p50/p95/p99 completion time and deadline misses. Success, failure, retry, timeout and cancellation rates by model version. Worker errors, poison messages and dead letter queue volume. Webhook delivery failures, repeat deliveries and polling load. Result storage failures, orphaned outputs and expired artifacts. Per tenant fairness and cost per successful completed prediction. Monitor SUCCEEDED only when the result has been committed and is actually readable. Correlate every log with the same job ID, tenant scope, model version and tracing ID . When to use—and not use—async inference Use async for long running document, image or video analysis, large payloads, bursty prediction traffic, expensive models that can wait in a queue, or workflows whose user can return later. Avoid async when a transaction requires an immediate prediction before proceeding (for example, a hard real time checkout decision), when the client cannot track completion, or when the overhead of queues and status storage is unjustified for a tiny low latency model. Distinguish the modes : Synchronous online: One request stays open until a prediction response or timeout. Asynchronous request driven: One request creates a job; caller checks later or gets notified. Scheduled batch: A pipeline scores a known dataset on a schedule or source data event, without requiring an individual API job per input record. Streaming: Event processing architecture with continuous ingestion, not just a queue of finite one off requests (Question 118). Amazon SageMaker Asynchronous Inference demonstrates a managed approach: the caller supplies an S3 input reference, the service queues inference and writes results to S3, with optional SNS notification. This is an example of a platform specific design; generic HTTP services can instead use their own queues, databases and workers. Common mistakes Returning HTTP 202 and treating it as proof that inference succeeded. Acknowledging jobs before durable status and enqueue state exists. Keeping job status only in API process memory. Using raw job IDs as access tokens. Re running duplicate client submissions without idempotency checks. Ignoring queue waiting time when promising a job completion SLA. Allowing infinite retries of invalid or permanently failing requests. Sending callback webhooks to arbitrary user provided internal network addresses. Marking a job completed before the output artifact is durable. Using one unbounded queue where a noisy tenant can starve everyone else. Confusing asynchronous request handling with automatic batch dataset processing. In short: Design asynchronous inference as a durable job protocol : validate and accept, return 202 and a job ID, process through bounded workers, persist status and results, handle duplicate submissions and failures safely, and measure the full time from acceptance to result availability.

What is a streaming inference architecture?

A streaming inference architecture continuously consumes events, prepares features, applies a deployed machine learning model, and publishes predictions with low end to end delay. Unlike a request driven asynchronous API, the system operates on an ongoing event stream rather than creating a separately polled job for each client request. Unlike scheduled batch scoring, it does not wait for a complete daily dataset before processing. Question 117 covered HTTP 202, request IDs and queued individual prediction jobs . Questions 115–116 covered serving mode selection and batch job integration. This lesson focuses on unbounded streams, event time correctness, state, throughput, scaling, backpressure and fault recovery . Beginner friendly example: real time delivery delay alerts A food delivery company continuously receives GPS updates, delivery status events and traffic alerts. It needs to estimate whether active orders will be late and notify operations while there is still time to intervene. A typical event looks like: The stream processor consumes the event, joins it with the latest valid order state, derives features such as recent travel speed, invokes a pinned delay risk model, and emits an alert when P(late) 0.80. A downstream consumer then updates a dashboard or triggers an appropriate alert. Avoid storing unnecessary location data in logs, and enforce retention and access restrictions. Core flow: Kafka is a durable event log broker; Apache Flink or Kafka Streams can transform, join and aggregate incoming events; inference may execute inside the stream processor or call a separately scaled model serving service. Stateless and stateful inference Stateless inference: Every event contains all necessary features. Example: classify each incoming text event using its content and language. The prediction itself needs no memory of previous events, although deduplication and output handling may still be stateful. Stateful inference: A prediction depends on multiple events for the same entity. Example: compute driver speed over the last five minutes, delivery distance remaining or the number of traffic alerts over the last ten minutes. A keyed state store maintains these aggregates and joins them to the new event. Be explicit about what is stateful: Feature state: Sliding window counts, recent positions, event time joins and last valid status. Model state: Loaded model weights and model version, sometimes updated through a controlled model release mechanism. Delivery state: Input offsets, processed event IDs, committed output versions and checkpoint metadata. Don't confuse stateful stream processing with recurrent or stateful ML models. A conventional gradient boosted model can consume features produced by a stateful stream processor. Event time versus processing time Event time is when the underlying business event occurred. Processing time is when a system actually handles it. Network interruptions and device buffering can make them different. Suppose GPS event A happened at 10:00:05 but arrived at 10:00:20 . Event B happened at 10:00:15 but arrived at 10:00:16 . Arrival order: B, A . Event time order: A, B . If a five minute speed feature relies on event order, processing the later arriving A as though it happened after B could produce incorrect state. Keyed event time windows, timers, version aware updates and late event handling help preserve historical semantics. Watermarks are an estimate of event time progress. A watermark of 10:05:00 indicates the processing system expects no more on time events at or before that timestamp. It is not an absolute proof that late records will never arrive. Configure a permitted lateness and correction policy. For a window [10:00,10:05), a watermark passing 10:05 may cause a window result to be emitted. A delayed 10:02 event arriving afterward can be: Dropped with an audited late event count. Routed to a side output for investigation. Included in a corrected/update result when allowed lateness and sink semantics support it. An event time window aligns with the historical event timestamps, while a processing time window depends on when the worker received or processed records. This matters for feature reproducibility and avoiding leakage. Window types and inference features Common patterns: Tumbling window: Fixed non overlapping five minute buckets; simple aggregation. Sliding window: Five minute window recalculated every minute; smoother signals with more state/compute. Session window: Group driver activity until an inactivity gap occurs. Per key running state: Last known position or active delivery status, often using event time ordering. For delivery delay inference, a sliding window could count recent traffic events, average route speeds and aggregate stops. Use time to live for state associated with finished deliveries and event time aware cleanup rules so state does not grow indefinitely. Numerical throughput and capacity example Suppose the platform receives 3,000 GPS events per second at peak. A stream processing/inference task handles 600 events per second at the required p95 latency under representative testing. Theoretical tasks = ceil(3000/600) = 5. Target 75% utilization to absorb spikes: effective throughput per task = 600 × 0.75 = 450 events/sec. Planned parallel tasks = ceil(3000/450) = 7. Seven tasks have a nominal capacity of 7 × 600 = 4,200 events/sec, but a 75% utilization planning capacity of 3,150 events/sec . These are planning estimates, not latency guarantees : hot delivery IDs, network I/O, joins, checkpoint overhead and model serving calls can create bottlenecks. Test skew and tail latency. Backlog example: Suppose an incident limits processing to 2,000 events/sec for 10 minutes while arrivals remain 3,000 events/sec . Deficit = 3,000 2,000 = 1,000 events/sec. Backlog growth = 1,000 × 600 = 600,000 events. After capacity returns, suppose processing can sustain 4,000 events/sec , while new arrivals remain at 3,000/sec . Net drain = 1,000 events/sec , so clearing 600,000 events requires 600,000 / 1,000 = 600 seconds = 10 minutes, ignoring changing traffic, queue overhead and other disturbances. The system can appear healthy in terms of worker uptime while producing ten minute old predictions , which defeats the business goal. Measure event time to prediction latency and consumer lag, not just raw inference execution time. End to end latency budgeting A realistic latency SLO measures prediction published at event time, with care for device clock skew and timestamp quality. It consists of: Device/broker transport and ingestion delay. Queue/broker lag. Deserialization, validation and feature enrichment. Waiting for event time windows or late data tolerance. Model execution (local or remote). Sink commit and downstream delivery. Suppose the target is p95 below 2 seconds for alert producing GPS events. A design that waits five minutes for a window to close cannot meet this target for all events. Instead compute rolling incremental features and emit provisional predictions continuously, correcting them later if required. Keep separate objectives for event processing latency , watermark lag , inference duration , sink publication latency and end to end freshness . Deploying the model inside or outside the stream engine Embedded inference: A model is loaded in each processing task. This avoids network calls, but uses task memory and makes model deployment/rescaling more tightly coupled to Flink jobs. Good for compact CPU friendly models. External inference service: Stream tasks call a dedicated serving tier. This lets GPU models scale independently, but adds latency, request failures, service quotas and backpressure. Use bounded concurrency, backoff and asynchronous I/O when supported, without dropping event IDs or violating ordering policies. Microbatch inference: Collect a small bounded set of events for efficient tensor operations, then publish results. Better accelerator utilization may add buffering latency. For all approaches, pin the model version for a defined processing epoch and include it in predictions. A rolling model upgrade can otherwise mix versions across parallel tasks. Use a controlled barrier, epoch marker or version stamped routing policy and record transitions. Backpressure and elasticity Backpressure happens when downstream processing is slower than upstream production. In a bounded stream processing pipeline this propagates to earlier operators or increases broker backlog. Likely causes: A slow model server or GPU saturation. Expensive window joins or oversized per key state. An overloaded database/output sink. Hot partitions, network congestion or checkpoint delays. Operational responses: Scale tasks and partitions when the bottleneck supports it. Increase serving capacity or use bounded inference batching. Reduce expensive remote feature lookups with permitted caching. Restrict concurrency to prevent overwhelming the model server. Apply producer rate controls, load shedding or degrade gracefully only under an explicit business policy . Track oldest event lag and stale prediction counts. More workers don't help if a single Kafka partition, hot key or external model endpoint imposes a hard throughput bottleneck. Plan sufficient broker partitions and correct keyed state partitioning. Fault tolerance and checkpoints A stream processor such as Flink coordinates checkpoints of: Source positions or offsets. Keyed operator state and event time timers. In flight data where supported by the checkpoint mode. Sink transaction state or identifiers needed for consistent output. On worker failure, the job restores the last completed checkpoint and replays events from the recorded source positions. For example, with a 60 second checkpoint interval , a worker that fails 45 seconds after a checkpoint may need to replay approximately 45 seconds of events . Recovery time additionally depends on checkpoint restore and processing speed; the true replay span can be larger depending on completed checkpoint timing. Streaming checkpoints protect processor state and input position , not necessarily the training optimizer state from Question 108. A reliable inference stream must also retain the pinned model artifact/version and feature enrichment dependencies needed to process replayed events consistently. At least once versus exactly once: the important boundary At least once delivery: Input events may be replayed after failure. Without deduplication, a prediction or notification might be emitted twice. Exactly once state consistency: A stream engine can restore internal state as though replayed records had been processed once at a logical checkpoint boundary. End to end exactly once effects: Also requires compatible sink transactions or idempotent writes and correct coordination with the source. An engine level guarantee does not mean every external HTTP request, email, database insert or business action is automatically executed once. For example, Kafka transactional producers and exactly once capable processors can support consistent consume process produce pipelines to Kafka with the correct configuration. But a stream worker that calls an arbitrary SMS API twice after checkpoint replay can still send duplicate messages. For externally visible risk alerts, use a stable event or decision ID such as delivery id + source event id + model version + rule version. Store/deduplicate published decisions, or use a transactional outbox to separate inference from the one time business action. Define whether an event can legitimately lead to multiple revised predictions. Checkpoint and sink trade offs More frequent checkpoints can reduce replay work, but increase coordination, storage and I/O overhead. Exactly once transactional sinks may delay visible output until the transaction commits. Unaligned checkpoints can sometimes ease recovery under backpressure but add checkpoint complexity and storage of in flight data. For a stream with large state , rescaling and restoring may take minutes even if single event inference normally takes milliseconds. Measure recovery time objective , checkpoint duration, restore failures, state size and data lag after recovery. Avoid assuming checkpoint interval alone determines maximum lost work: a failed/incomplete checkpoint and slow barrier alignment change the actual recovery position. Practical pseudocode (framework independent) This is illustrative pseudocode , not a directly executable Flink application. An actual Flink/Kafka Streams implementation uses framework managed keyed state, timers, watermark strategies, appropriate checkpoint configuration and an output sink that satisfies the chosen delivery semantics. The ordering of emit and mark processed in ordinary application code cannot alone guarantee exactly once effects after failures. What to monitor Input: Events/sec, broker partition lag, oldest unprocessed event, schema errors and rejected events. Freshness: Event time lag, watermarks and p50/p95/p99 event to prediction latency. Operators: Busy/backpressured/idle time, queue depth, hot key skew, task restarts and resource use. State: Key count, state bytes, state TTL expiry, restore duration and checkpoint completion/failure. Model: Pinned version, prediction distributions, feature missingness, inference latency, errors and drift after labels arrive. Output: Sink publish failures, duplicates, dropped or late events, transactional commit delay and downstream consumption lag. SLO: Percentage of valid events with predictions published within deadline, and accuracy/coverage once ground truth is known. Alert when the stream is running but stale : healthy containers do not imply useful real time predictions. Trade offs and common mistakes Stateless processing: Simpler and cheaper but cannot represent recent behavior without external state. Stateful feature windows: Better temporal signals, at the cost of memory, checkpoint overhead and late event semantics. Embedded models: Lower network latency, but model loading and GPU needs couple inference to processor lifecycle. Remote model serving: Independent scaling and model specialization, but harder to meet strict latency and failure budgets. Higher parallelism: More throughput until partition/key skew or external services become the bottleneck. Long lateness window: More complete event time results, but delayed final decisions or more correction events. Exactly once transactional pipelines: Stronger consistency with additional complexity and possible output delays. Blind retries of external side effects: Can duplicate alerts even when stream state itself recovers correctly. In short: Streaming inference is a continuously running event to prediction pipeline . Design the broker, stateful feature processing, event time windows, model execution and output sink as one reliability contract. Scale for peak throughput, measure event to prediction freshness, recover from checkpoints, and treat exactly once promises as end to end claims that must be verified at every output boundary.

How do online, batch, and streaming serving architectures differ operationally?

Online, batch and streaming inference can use the same trained model, but they require different operating procedures after deployment. Online teams keep request serving endpoints healthy, batch teams guarantee complete prediction datasets by deadlines, and streaming teams keep continuous event processing correct and sufficiently fresh. The earlier lessons covered choosing a serving mode (Question 115), batch execution mechanics (116), request driven async APIs (117), and streaming fundamentals (118). This lesson is about day 2 ownership: operating, upgrading, scaling, recovering and auditing each mode over time . Practical example: one delivery risk model, three operations teams A food delivery company owns a registered risk model late risk/27. Three consumers use it: Online checkout service: Calls an endpoint to obtain a delivery risk estimate before confirming an order. Nightly batch planning: Scores 1 million potential deliveries for the next day and publishes a complete dataset. Continuous driver event monitoring: Updates risk from incoming GPS and order status events and generates live operational alerts. Although the model weights might be shared, serving contracts cannot be treated as interchangeable . A healthy batch job does not guarantee a responsive API; an API with 99.9% uptime does not guarantee streaming predictions are current; an active stream processor does not prove daily bulk output completeness. Operational comparison by responsibility (pointers, not a table) Online serving Primary owner: API/model serving on call team, coordinated with application, platform and feature store owners. Input arrival: Live requests with bursty traffic and an interactive request/response contract. Runtime topology: Load balancer or API gateway, serving replicas, feature dependencies and observability. Primary SLO: Availability, error rate and end to end p95/p99 request latency. State: Usually stateless serving replicas plus external feature/cache state; model weights remain loaded. Scaling: Replicas, concurrency, GPU utilization, admission limits, warm pools and autoscaling lag. Release: Rolling, blue green or canary deployment with rapid traffic rollback. Recovery: Fail over to healthy replicas or a prior serving revision, with bounded request retries. Cost: Often includes provisioned or warm idle capacity plus peak headroom. Batch serving Primary owner: Data platform/pipeline team, partnering with model and downstream analytics owners. Input arrival: Committed, versioned input partitions on a business schedule or data ready event. Runtime topology: Job scheduler, ephemeral scoring workers, output storage and consumer handoff. Primary SLO: Correct, complete predictions published by a specified deadline. State: Input snapshot ID, logical interval, partition receipts, output manifest and task progress. Scaling: Worker count, partition size, executor resource quota and backlog of pending windows. Release: Pin a model version per logical run; promote a new model on a chosen future run. Recovery: Retry incomplete partitions and publish a new atomic output revision; do not expose partial files. Cost: Compute during scoring windows, storage and data movement, plus retries. Streaming serving Primary owner: Stream processing/platform team with model serving, broker and downstream alerting owners. Input arrival: Unbounded event streams, often partitioned by an event key. Runtime topology: Message broker, stream processor, keyed state/checkpoints, model execution and event sink. Primary SLO: Event to prediction freshness, throughput, lag, checkpoint success and output correctness. State: Source offsets, keyed/window state, timers, model version and sink delivery state. Scaling: Consumer parallelism, broker partitions, keyed state resharding and inference capacity. Release: Controlled upgrade of processing logic, state schemas and pinned model epochs, sometimes with savepoints. Recovery: Restore coherent stream state/source positions and replay, then reconcile duplicate external effects. Cost: Continuously operating broker/processors, state storage, checkpoint I/O and serving resources. Each system should have a clearly named service owner, runbook, alert severity, change approver, dependency owner and recovery objective . Shared model ownership does not remove infrastructure specific responsibilities. Numerical example: three different operating SLOs Assume: Online checkout peaks at 200 requests per second . Required p95 response latency ≤ 250 ms . Batch planning must publish 1,000,000 scores by 06:00 UTC . Streaming ingests 2,000 events/sec with p95 event to prediction delay ≤ 3 seconds . These are three different types of promises. Online capacity: Suppose a replica sustains 50 RPS while meeting its tested latency. At a targeted 70% utilization, planning capacity is 50 × 0.70 = 35 RPS/replica. Needed replicas = ceil(200/35) = 6. Six is a capacity estimate, not proof of p95 latency under bursts. Batch capacity: Suppose four workers together score 20,000 rows/minute . Compute time is 1,000,000 / 20,000 = 50 minutes. With 15 minutes of startup, validation and output publication, estimated completion is 65 minutes . Starting at 04:45 gives an estimated finish at 05:50 , only 10 minutes of deadline margin , so retries or skew may breach the SLO. Streaming capacity: Suppose processors sustain 2,500 events/sec while arrivals are 2,000/sec . Utilization is 2,000/2,500 = 80%. If an outage reduces sustained processing to 1,500/sec for 6 minutes , backlog increases by (2,000 1,500) × 360 = 180,000 events. At restored capacity 2,500/sec, with arrivals still 2,000/sec, net catch up is 500/sec and draining takes 180,000/500 = 360 seconds = 6 minutes in an idealized model. The service may be running but late, so its freshness SLO can fail throughout catch up. Operational lesson: Online errors happen at a request boundary; batch failures delay an entire publication; streaming failures can create a growing lag that persists long after processes restart. Day 2 release management differs Online rollout Build a validated immutable serving image plus model/feature contract. Start new replicas, load and health check model dependencies. Shift a small share of live traffic to the new version. Compare errors, p95/p99 latency, resource consumption and prediction behavior. Increase traffic gradually or restore the previous deployment. Treat a traffic rollback as a serving action, not as deletion of the new registry version. Batch rollout Select a specific registry version at the beginning of a logical scoring interval . Keep the version fixed across all scoring partitions. Validate counts, output schema and prediction distribution. Commit a result manifest only after complete verification. If a model bug is discovered, re score affected windows as a new output revision and promote it deliberately. Streaming rollout Pin model identity and feature/state schema in the running job. Plan operator state compatibility and source offset continuity. For stateful job changes, use compatible savepoint or checkpoint migration where supported; test restores before production. Define a model version transition boundary and include version in outputs. Validate no event gaps or unintended duplicate side effects during cutover. A streaming job rolling update that changes keyed state structure may require a migration rather than a simple replica replacement. A batch job with two models mixed within one output snapshot is inconsistent even if every partition succeeded. An online deployment with new model weights but incompatible live feature schema can fail during a canary despite passing offline evaluation. Operational consistency: what does “correct” mean? For online: Correctness at request time means a versioned request is validated, current enough features are used, and its response belongs to the intended model. Retrying a request may produce another response if inputs change; define idempotency where externally visible side effects exist. For batch: Correctness for the logical interval means all eligible records are scored with the pinned model and input snapshot, with no unexplained duplicate/missing keys and one committed output revision. For streaming: Correctness over an event timeline requires event time handling, state consistency, source positions and sink semantics. Exactly once internal checkpoint restoration does not imply an email, HTTP callback or other external action happened exactly once. Deduplicate or transactionally coordinate side effects. Model version consistency applies to all three, but the lifetime differs: individual request, scoring run, or processing epoch. Scaling and cost: pay for different bottlenecks Online: Sustained traffic often justifies always ready replicas; strict tail latency requirements may require spare capacity. Serverless options trade idle cost for potential cold start limits. Batch: Add workers or partitioning to meet a deadline; after job completion compute may deallocate. High parallelism can still be limited by storage reads or skew. Streaming: More partitions and task slots can raise throughput, but large keyed state, checkpoint barriers, a hot key or a single remote model endpoint may limit scaling. Illustrative prices, not provider quotes : Two 24/7 serving instances at $0.50/hour over 30 days: 2 × 24 × 30 × 0.50 = $720 in compute. Four batch workers at $0.50/hour for 2 hours/day over 30 days: 4 × 2 × 30 × 0.50 = $120 in compute. Three continuously running streaming workers at $0.50/hour : 3 × 24 × 30 × 0.50 = $1,080 in worker compute. These scenarios have different capacity, workloads and supporting costs. They do not prove that one mode is always cheaper. Count broker, network, object storage, state, logging, failure retries, model inference calls and operational labor before comparing true total cost per business outcome. Azure ML documentation makes an important platform distinction: online endpoint resources are commonly provisioned with the deployment, while batch deployment compute can be allocated for the job and released afterward. Actual bills depend on service configuration and cloud provider. See official Azure and SageMaker references. Failure recovery and escalation runbooks Online incident: p99 latency spikes 1. Check feature dependencies, request queues, saturation, latency by model version and recent rollout. 2. Reduce traffic to unhealthy/new versions; scale or shed overload safely. 3. Preserve request IDs and the serving version needed for investigation. 4. Verify restored p95/p99 and error rates before incident closure. Batch incident: scores not published by deadline 1. Identify the logical window and source/model identities. 2. Inspect failed partitions, upstream readiness, worker failures and output reconciliation. 3. Retry only incomplete safe partitions or start a controlled new revision. 4. Publish once all requirements pass, record whether the SLA was missed, and notify consumers of availability or fallback status. Streaming incident: output lag growing 1. Compare source arrival rate, input/consumer lag, processing rate, watermark lag and sink errors. 2. Find the bottleneck: partition skew, model calls, checkpoint failure or downstream backpressure. 3. Restore from a valid checkpoint/savepoint when needed and replay safely. 4. Track catch up until output freshness recovers; verify duplicates or missed events do not violate contracts. Avoid a universal “restart the service” playbook. Restarting online replicas may briefly reduce capacity; restarting a batch job may duplicate outputs; restarting streaming can replay events and repeat side effects. Monitoring and incident response dashboards Online dashboard: request count, error rate, p50/p95/p99, timeouts, saturation, model versions, warmup and feature store latency. Alert on user visible SLO violations, not CPU spikes alone. Batch dashboard: expected/complete partitions, scoring run ID, logical date, rows scored, quarantine rates, publication deadline, last committed output revision and failed retries. Alert when a required output will not meet its deadline. Streaming dashboard: input/processed events, source lag, watermark lag, event to output latency, backpressure ratio, keyed state size, checkpoint age/duration, sink commits and restart count. Alert on stale predictions even if workers report healthy. Common model quality dashboard: prediction distribution, feature completeness, data/feature drift, delayed label performance, version provenance and segment performance. Labels may arrive on different schedules across serving modes; interpret model quality with appropriate time alignment. Hybrid operational ownership Real systems often combine modes: Batch scores the daily risk of every planned order. A streaming processor updates route context as GPS events arrive. An online endpoint combines precomputed risk with fresh information at checkout. To keep this coherent: Establish a shared model/feature schema and version catalog . Record the source timestamp and freshness budget of each precomputed or streamed feature. Define which layer is the system of record for each prediction or alert. Prevent the same business action from being triggered independently by batch and stream without an explicit deduplication rule. Set clear ownership for cross mode incidents and correlated tracing IDs. Run parity checks on comparable records to detect offline/online feature skew while acknowledging intentional freshness differences. For example, the online service should not silently assume that today's batch risk score is valid if the newest batch failed. It must enforce a maximum age and choose a documented fallback. A practical operating contract For each deployed mode, maintain a short service contract containing: 1. Owner and consumer: Who publishes predictions, who uses them and who is on call. 2. Version and input: Approved model identity, feature schema, event or snapshot version and release boundary. 3. Primary SLO: Request latency, batch publication deadline or event to prediction freshness. 4. Capacity: Peak RPS, batch records/time or streaming events/sec and resource headroom. 5. Failure semantics: Retry scope, idempotency, output visibility and rollback/replay strategy. 6. Data correctness: Freshness, completeness, duplicate handling, version parity and audit linkage. 7. Monitoring: Dashboards, alerts, runbooks and incident review criteria. 8. Cost: Compute, storage, state, data movement, retries and cost per useful prediction. 9. Change policy: Canary, future run promotion or stateful stream migration. Do not assume a platform's autoscaling or health check alone enforces this contract. Common operational mistakes Applying one latency SLO to batch publication and streaming event freshness. Alerting on “workers down” while missing a running but stale stream. Promoting a model alias midway through a batch run. Rolling out a new stateful stream processor without restore compatibility tests. Treating internal exactly once state guarantees as automatic exactly once external notifications. Measuring batch job completion without verifying the consumer visible committed output. Equating online uptime with successful model responses and valid feature values. Assuming CPU/GPU replicas alone solve hot key or remote service bottlenecks. Calculating cost from worker runtime but ignoring broker, state, idle capacity and storage. Failing to agree on who owns an incident when batch, stream and online outputs are combined. In short: Operate online inference as a low latency service , batch inference as a deadline bound data product , and streaming inference as a continuously stateful event processing system . Keep model identity and business correctness consistent, but tailor release management, incident recovery, monitoring and cost controls to each operating mode.

How do you package a registered model into a reproducible production inference deployment?

A reproducible inference deployment is a release whose exact model bytes, preprocessing code, runtime image, dependency versions, input/output contract and configuration can be identified, tested and redeployed without depending on a mutable “latest” pointer. An approved model in the registry is necessary but insufficient: the serving runtime must reliably load and execute it. Question 119 compared online, batch and streaming operations. Questions 40–44 covered basic containers, while Questions 85–89 covered registry versioning and triggers. Here we connect an approved model to an immutable, deployable inference release ; later lessons can examine traffic rollout and CI/CD policies in detail. Beginner friendly story: same model, different result A delivery company approves model late risk version 27. In staging the risk score for a test delivery is 0.72. In production it becomes 0.61 even though both services claim to use “version 27”. Investigation reveals staging used preprocessing package 2.4, production used 2.2, and one server fetched the mutable production model alias at startup. The model name alone did not uniquely describe the actual inference system. The fix is a release manifest , not merely retraining: Model: registry version 27 and immutable artifact digest, for example sha256:model abc (illustrative). Inference image: pinned image digest, for example sha256:image def (illustrative). Preprocessing code: Git commit and package version. Runtime: locked Python/framework/native library versions and target CPU/GPU architecture. Contract: schema version, units, required/optional fields, output interpretation and tested request fixtures. Runtime settings: external feature endpoint version, timeout, batch/concurrency limits and secret references; never embed secrets. Evidence: evaluation approval, artifact scan, integration test results, deployment environment, owner and rollback reference. A deployment is reproducible in configuration and executable artifacts ; byte for byte identical outputs across different hardware are not automatically guaranteed by pinning alone. Floating point kernels, nondeterministic operations, remote features and changing input data can still change predictions. Step by step production release 1. Resolve approval to immutable identity. CI receives a registry event, checks model approval and resolves the exact registered version to verified artifact bytes. Record the version and checksum. Never resolve a mutable alias separately on different replicas. 2. Build or select a compatible inference runtime. Bundle serving code, preprocessing and explicitly locked dependencies, or retrieve model artifacts by a pinned digest from a trusted store at startup. Reject missing digests or incompatible formats. 3. Specify the prediction contract. Define required columns, types, units, null handling, output names, model signature and explicit error behavior. A valid schema is necessary but does not guarantee semantic feature correctness. 4. Run artifact and security checks. Verify the approved provenance, build image and model checksums, dependency/license policy, architecture and necessary permissions. Treat serialized models from untrusted sources as executable code risk. 5. Test the real deployed path. In an isolated environment, load the artifact and call the actual inference handler with known examples, malformed inputs and missing feature cases. Compare toleranced predictions to golden fixtures and test dependency failures. 6. Publish an immutable release manifest. Store model digest + image digest + schema/config version + test evidence together. Sign or attest where your organization requires it. Tag names alone are not immutable; deploy by digest. 7. Deploy into a staging environment with production like dependencies. Confirm startup, readiness, warmup, resource limits, structured logs and timeouts, then run smoke and load checks. Readiness means this revision can serve requests , not simply “the HTTP process is alive.” 8. Promote that same release identity to production. Environment specific secrets and endpoints may change under controlled configuration, but do not quietly rebuild a different image or substitute model files. Record who promoted it. 9. Observe and retain rollback material. Log release ID, request trace, model/feature versions, errors, latency and resource use. Preserve the last known good image, model and compatible configuration. The stages are conceptually applicable to online endpoints, batch scoring workers and stream processors. For batch, pin the release for the whole logical scoring run; for streaming, also coordinate the release with stream state compatibility and checkpoint boundaries. Numerical example: why release pinning matters Suppose 4 serving replicas each process 25 requests/sec, for 100 RPS total . Two replicas restart and fetch mutable model alias production, which now points to version 28 while the other two continue to run version 27. Version 27 returns 0.72 for a fixed regression example; version 28 returns 0.66. At a balanced 100 RPS, about 50 requests/sec would reach each version group. Over 10 minutes, that is roughly 30,000 requests per version out of 60,000 total, ignoring traffic skew and failures. A client with repeated requests can receive inconsistent scores even if every server reports healthy. The fix is to roll out a single manifest that specifies model version 28, the tested image digest and schema. The fleet can still intentionally contain two versions during a controlled rollout, but the split is explicit, observable and reversible rather than an accidental consequence of restarts. Numerical example: memory and startup budget Assume a 600 MiB in memory model, 350 MiB interpreter/framework overhead, 250 MiB preprocessing/cache overhead and 200 MiB peak request buffers per replica : Estimated resident footprint: 600 + 350 + 250 + 200 = 1,400 MiB. With a planning margin of 25%: 1,400 × 1.25 = 1,750 MiB; a 2 GiB limit is a reasonable initial test candidate, not a guaranteed safe limit. Four warm replicas consume roughly 4 × 1.4 = 5.6 GiB of modeled footprint before node overhead. If each cold replica takes 45 seconds to fetch, deserialize and warm the model, scaling decisions must account for that delay; a liveness probe that restarts the process after 10 seconds creates a restart loop. Measure real peak memory, startup percentiles and concurrency under load. Model artifact size on disk is not necessarily the same as resident memory. Set an appropriate startup probe and mark readiness true only after model loading and essential dependencies are checked. What belongs in image versus model store? Bundle model bytes in the image when modest artifact size, air gapped deployments or guaranteed self contained startup is more valuable than image size. Trade off: large image transfers and duplicated layers during frequent model revisions. Fetch a pinned model artifact at startup when models are large or updates are frequent. Trade off: artifact store availability, checksum verification, cold start latency, storage permissions and partial download handling. Download to a temporary path, verify digest and atomically make the artifact available. Avoid mutable paths such as models/latest.pkl or pulling an unpinned container tag at restart. For both patterns, the manifest must resolve every runtime dependency to an audited identity. Health checks and failure scenarios Startup: The model is still downloading or warming. Use a sufficiently generous startup budget; do not treat loading as a fatal liveness failure. Readiness: The process is alive but model load, required configuration or a critical dependency failed. Keep it out of request traffic; avoid unnecessarily declaring all replicas unready for a transient optional downstream dependency. Liveness: The process is irrecoverably stuck. Restart only when restart is likely to help, rather than for ordinary feature store timeouts. Bad artifact: A checksum mismatch or incompatible serialization must fail closed and report the release ID and provenance. Schema mismatch: Reject incompatible requests clearly; never silently rearrange or default semantically different features. Rollback: Restore an entire compatible release bundle , not just previous weights, when preprocessing/config has changed. For a live API, test HTTP/gRPC request correctness and tail latency. For batch, check the same release over fixture partitions and output manifests. For streaming, validate event decoding, stable model identity and state transition constraints before resuming from saved positions. Monitoring and reproducibility evidence Track: Deployed release manifest ID, model digest, image digest and signature/schema version by instance. Successful/failed startup, artifact fetch duration, checksum failures, readiness transitions and warmup duration. Request validation failures, feature dependency errors, latency, memory and prediction drift by revision. Release to environment mapping and staged test evidence. Ability to launch a prior release from retained immutable inputs; periodically exercise rollback and disaster recovery. A successful Docker build proves only that an image was produced. A successful health endpoint proves only what that probe checks. The production readiness gate needs an actual prediction round trip using the exact release artifacts. Trade offs and common mistakes One image per model: Easy to audit and reproduce but can increase build/pull cost. Generic serving image plus pinned model: Promotes runtime reuse but needs strict model/runtime compatibility checking. Strict deterministic output: Useful for regression tests, yet use numeric tolerances where legitimate accelerator or kernel differences occur. Heavy dependency pinning: Improves repeatability but requires scheduled security upgrades and rebuilds. Feature service as an external dependency: Pin its contract and freshness policy; a reproducible container cannot freeze real time external data. Watch for: model alias resolved independently by each replica; deploying a mutable image tag; missing preprocessing/version coupling; loading opaque pickle from an untrusted source; readiness before model warmup; leaking credentials into images; relying on only notebook evaluation; losing last good artifacts; and “rollback” that changes weights but not the compatible feature schema. Deployment acceptance checklist Approved immutable model version and checksum. Verified image digest, locked dependencies and inference code revision. Explicit model signature, pre/postprocessing contract and feature source assumptions. Test fixture with expected output and tolerances; rejected malformed requests. Startup/readiness/liveness and load tested resource budgets. Environment config/secrets externalized and traceable. Published release manifest, promotion evidence and verified rollback target. Takeaway: A production inference release is model + executable runtime + feature contract + controlled configuration + verifiable evidence . Treat that combined identity as the unit you test and deploy, so later CI/CD and canary rollout steps can move a trusted release instead of assembling an uncertain one at runtime.