MLOps
MLOps applies software engineering and operations discipline to machine learning. It covers reproducible training, model registries, deployment, monitoring, governance, and incident response.
How do offline and online feature stores differ operationally?
An offline feature store is optimized for retrieving large volumes of historical feature data for training and batch scoring. An online feature store is optimized for quickly retrieving the latest available feature values for an entity during live inference. They commonly share definitions but have different latency, throughput, storage, freshness and failure requirements. Question 90 introduced what a feature store does. Here we examine how its two retrieval paths operate . Detailed historical point in time join algorithms follow in Question 92. Beginner friendly end to end example A delivery company predicts whether orders will arrive late. It uses restaurant avg delay 30d and customer orders 7d. Offline path — model training 1. The company stores historical restaurant and order events in warehouse/lake tables. 2. A data job computes feature histories with entity IDs and timestamps. 3. A training job asks for historical features for 500,000 past delivery orders . 4. The offline retrieval engine scans or joins historical data and produces a training dataset. 5. Training can take minutes or longer for that dataset: the priority is correctness and efficient bulk processing, not milliseconds per row. Online path — live prediction 1. The same feature definitions identify what the online service requires. 2. A materialization job or streaming ingestion updates the latest feature values for each restaurant/customer in a low latency store. 3. A new order for customer C42 at restaurant R10 arrives. 4. The prediction service retrieves recent values for those two entity keys and combines them with request time information such as the trip distance. 5. The model generates its response within the end to end serving latency budget. Not every platform persists all calculated features in a separate offline database: some feature store systems query existing warehouses as their offline source. Practical numerical comparison Assume: Training requires 500,000 historical entity time rows . A warehouse query completes in 4 minutes (illustrative, not a guaranteed performance level). The online service receives 200 inference requests per second , and the online feature lookup takes 8 ms at P95 under measured load. Then: Offline bulk retrieval throughput: 500,000 ÷ 240 ≈ 2,083 rows/second on average for that query . Online lookup traffic: 200 × 60 = 12,000 requests/minute. At 200 requests/second and 8 ms average latency if it were the mean rather than P95 , expected concurrent lookups would be 200 × 0.008 = 1.6 by Little's law. Do not apply that concurrency estimate to P95 latency directly : P95 is a percentile, not an average. These statistics describe different workload types. A warehouse completing millions of rows in minutes can be excellent for training yet unsuitable for synchronous user facing retrieval. Operational differences Storage and query pattern Offline: Warehouses, lakes, Parquet files or distributed analytical engines; scans, joins and time based historical queries over large datasets. Online: Key value or other low latency databases, indexed by entity keys; retrieves a small number of current values per prediction. An online store usually retains the newest values per entity/feature view, not the full historical event series. Historical versions generally remain in the underlying offline sources. Freshness Offline pipelines may update hourly or daily depending on the task. Online features may update through scheduled batch materialization or streaming writes, depending on latency requirements. A fast online response does not guarantee a fresh feature : 5 ms retrieval of a 2 hour old value is still stale. Materialization Materialization loads computed values into the online serving store, often from offline sources. For example, a scheduled job may move the latest eligible restaurant statistics to the online store every 15 minutes. Streaming ingestion may achieve a different freshness profile, but needs its own ingestion, event ordering and recovery controls. Consistency Aim for shared definitions, compatible types and agreed timestamps. Expect temporary offline/online differences due to refresh delays, event timing and corrections. Compare feature values for sampled entity/timestamp pairs and monitor discrepancies; the stores need not always be byte for byte identical at every instant. Reliability Offline failures can delay training or scheduled materialization; jobs can often be retried or backfilled. Online failures directly affect prediction requests and can cause latency spikes, missing features or serving errors. Define fallbacks for missing/expired features, and ensure the model was trained to handle those fallback patterns. Costs Offline costs commonly include data scans, historical storage and scheduled compute. Online costs commonly include provisioned capacity, high availability, RAM/storage, writes and low latency requests. Streaming updates, replication and materialization add operational costs that depend on throughput and freshness targets. Beginner friendly freshness calculation Suppose restaurant avg delay 30d was last successfully refreshed at 10:00 AM . A prediction arrives at 10:18 AM . Observed feature age: 18 minutes . Business freshness limit: 15 minutes . The feature is 3 minutes older than allowed . The system should apply its defined policy: reject the prediction, use an approved fallback or another eligible feature value, and record the degraded state. Do not silently treat a stale value as current. Age calculations must use compatible event/ingestion timestamps and clock conventions. Example Feast retrieval APIs With a configured Feast repository and defined feature view: This example is illustrative , not standalone executable: historical orders must contain valid entity keys and timestamps, restaurant stats must exist, and configured offline/online storage must be populated. In particular, merely declaring a feature view does not populate its online values. For materialization, Feast exposes the feast materialize workflow; the exact schedule, timestamp range and supported ingestion mode depend on your configured sources and infrastructure. Production checks 1. Offline: Verify historical retrieval completeness, dataset version, scan cost, job duration, and retries. 2. Online: Track lookup P50/P95/P99 latency, throughput, error rate, cache hit behavior and availability. 3. Freshness: Alert on time since latest successful feature update, separately from read latency. 4. Missing values: Measure missing/stale feature rate, and test fallback behavior. 5. Materialization: Monitor the last successful job, input to online lag, failed writes and duplicate/late updates. 6. Consistency: Periodically compare offline and online values where the timestamps make a fair comparison possible. 7. Capacity: Load test peak predictions and ingestion writes together, including store recovery scenarios. Common mistakes Running expensive historical warehouse joins inside a synchronous prediction request. Assuming an online feature store automatically contains every historical value. Confusing fast lookup latency with feature freshness. Expecting all offline to online updates to be instantaneous. Ignoring stale or missing values during training and then encountering them unexpectedly in production. Comparing offline and online snapshots from different times without accounting for lag. Paying for always on online serving when a purely batch workload only needs offline retrieval. In short: Use the offline path for historical breadth and bulk efficiency , the online path for current entity lookups and low latency , and a carefully monitored materialization/ingestion process to connect them. Align feature definitions, measure freshness independently from latency, and prepare for both batch and real time failures.
How is point-in-time correctness enforced in feature retrieval?
Point in time correctness means each historical training example receives only feature values that were valid—and, where the use case requires it, actually available—at the moment its prediction would have been made. This prevents future data leakage , where a model appears excellent in training because it sees information that production could not have known. Step by step numerical example A delivery company wants to predict a late delivery for restaurant R10 at 10:00 AM . Its feature is restaurant avg delay minutes. A two hour feature TTL is configured. Historical feature records for R10: 8:30 AM event time, available 8:32 AM , feature value 5 minutes . 9:40 AM event time, available 9:45 AM , feature value 7 minutes . 9:55 AM event time, available 10:20 AM (late arrival), feature value 9 minutes . 10:15 AM event time, available 10:17 AM , feature value 12 minutes . Step 1: Match entity. Only records for restaurant R10 qualify. Step 2: Apply the prediction timestamp. For the 10:00 AM prediction, no feature with an event timestamp after 10:00 AM is eligible; the 10:15 AM value 12 is excluded. Step 3: Apply the two hour TTL. Eligible event times must be within 8:00–10:00 AM . The 8:30 AM, 9:40 AM, and 9:55 AM events pass this event time age test. Step 4: Account for actual availability. The 9:55 AM event was not processed until 10:20 AM , so a real 10:00 AM inference service could not have used its value 9 . Under a strict availability aware historical reconstruction, exclude it. Step 5: Choose the newest remaining eligible value. The 9:40 AM event, available at 9:45 AM, is the most recent admissible record. The correct feature value is 7 minutes . What would go wrong? A naïve event time only join could select 9 (9:55 AM event), overstating information the live model actually possessed. A current value lookup might select 12 , which is outright future leakage. Two clocks: event time and availability time Feature event time: When the represented event happened or its feature value was measured. Availability/ingestion time: When the information became accessible to the relevant production feature pipeline. Entity timestamp: The historical decision/prediction moment for the example, sometimes called event timestamp in the entity DataFrame. Created timestamp: In feature pipelines, often used for record creation and deduplication. It is not automatically equivalent to production availability time. A sound rule for strict production replay is: Choose the most recent admissible feature, with a deterministic revision tie break. In some systems the production cutoff includes upstream data latency or a fixed feature publication delay; reconstruct the exact policy used at serving time. Feast point in time joins Feast's get historical features matches feature values to historical entity rows. For each entity timestamp, its offline retrieval searches backward for the latest matching feature event time , bounded by the FeatureView's TTL . Feast documentation explicitly notes the TTL is measured relative to each entity timestamp , not relative to the current query execution time. This is illustrative: restaurant stats must be an existing configured FeatureView with an entity join key, event timestamp column, source and an appropriately configured TTL; the offline data must be accessible. No fictional output is asserted. FeatureView TTL is typically defined using ttl=timedelta(hours=2). Source rows and timestamps determine the actual results. Important limitation: Feast's documented point in time join protects against using a feature event timestamp after the entity timestamp , and may use created timestamps to deduplicate duplicate feature rows. It does not by itself guarantee that late arriving historical records are excluded according to a production availability clock . If that distinction is critical, capture available at or ingestion/publication metadata and use versioned snapshots, availability aware filtering, or a custom offline retrieval process validated against serving history. Do not claim the default Feast join solves all forms of late arrival leakage. Why TTL matters TTL limits how far backward a join searches. For a prediction at 10:00 AM and TTL of two hours, a feature event from 7:45 AM is too old even if it was the latest available for that entity. The correct result may be missing/null , not the stale value. This prevents unbounded use of obsolete features. However, TTL is not a proof of actual freshness or availability: the retrieved event could have been ingested after the prediction. The SQL intuition: an as of join A simplified availability aware query for one example is: This uses timestamps assumed to share the same timezone, with available at representing actual availability in the relevant serving system . In production, use timezone aware types and apply this logic separately to every entity time row, often through a lateral/as of join or an optimized feature store engine. If two records share the same event timestamp, define how corrected values were published and which revision was available at the cutoff. Never use an arbitrarily current correction when building a supposedly historical training row. How to test for leakage 1. Future event test: Add a feature record at 10:01 AM and predict at 10:00 AM. The result must not change. 2. TTL boundary test: With a two hour TTL, verify a 7:59 AM value is excluded at 10:00 AM, while a qualifying 8:00 AM value is handled according to defined inclusive boundary rules. 3. Late arrival test: Add the 9:55 AM event whose available at is 10:20 AM. Strict availability aware retrieval at 10:00 AM must still return 7 , not 9 . 4. Entity isolation: Feature rows for restaurant R11 must not affect R10. 5. Tie/revision test: For duplicated event timestamps, verify deterministic revision selection without future corrections. 6. Missing value test: When no row qualifies, ensure a missing value and the documented fallback are used, rather than a future or stale value. 7. Replay test: Compare historical as of values to logged online feature snapshots for sample predictions; investigate mismatches from ingestion lag or data corrections. Operational safeguards Preserve immutable feature source snapshots, source watermarks and publication timestamps where needed. Keep training cutoff times, feature view definitions, query versions and entity row timestamps reproducible. Ensure labels are computed independently from future derived feature fields. Separate the target label observation window from the prediction time feature cutoff. Document timezone and clock skew handling, especially for distributed producers. Do not assume created timestamp always proves the record was available to a production service; confirm its precise source semantics. Common mistakes Joining the latest known value today onto a historical observation from last month. Checking event time but ignoring delayed arrivals that were unavailable at prediction time. Measuring a Feast TTL backward from today instead of from each entity timestamp. Returning stale values despite a configured freshness boundary. Comparing local times with UTC timestamps without normalization. Deduplicating corrected records without preserving what was historically visible. Assuming good offline validation accuracy rules out leakage. In short: Match by entity, restrict feature events to the prediction time window, enforce TTL, and—when accurate production replay requires it—ensure the feature was actually available by that time. Feast supplies event time point in time joins; availability time and late correction guarantees may need additional data engineering and validation.
How do feature materialization jobs move values into an online store?
Feature materialization is the process of copying or computing eligible feature values from a batch or historical source into an online store so a prediction service can retrieve them quickly. A materialization job is not a training job: its main output is updated serving time feature state. Practical example A food delivery model uses restaurant avg delay 30d to predict late deliveries. At 10:00 AM, the historical source has: Restaurant R10: 8 minutes, event timestamp 9:45 AM. Restaurant R20: 5 minutes, event timestamp 9:50 AM. Restaurant R30: 12 minutes, event timestamp 9:55 AM. A scheduled materialization job reads the eligible latest records for these restaurant IDs, validates their schema, and writes their values to an online key value store. A prediction for R20 can then retrieve 5 minutes without scanning a warehouse. At 10:15 AM, the source updates R20's feature to 7 minutes with an event timestamp of 10:10 AM. The next successful materialization updates only the R20 online value (alongside other eligible changes). The online store ordinarily retains the latest value per entity and feature , rather than every historical record. Feast's online store documentation describes this behavior. End to end workflow A practical sequence: 1. Define feature views. Specify entity keys, event timestamps, source columns and online store mapping. 2. Configure storage. Provide a reachable online store such as Redis or another supported backend, with appropriate credentials. 3. Choose the load window. For scheduled batches, materialize events from an agreed start to end time. An incremental process tracks the previous processed boundary. 4. Read and transform data. Retrieve eligible feature records and enforce validation rules. 5. Update entity values. Write values to the online store while respecting event time ordering and the backend's supported update semantics. 6. Verify success. Check ingestion counts, failures, freshness and sampled online reads. 7. Schedule the next run. Reconcile late arriving and corrected records according to a documented recovery policy. Numerical example: job cadence and lag Suppose a pipeline creates new restaurant statistics every 5 minutes , and a materialization job runs every 15 minutes . A value computed at 10:01 AM may not be loaded until the 10:15 AM run finishes. If that job completes at 10:17 AM, the value's age at availability is 16 minutes . A feature arriving just after a materialization cutoff may experience approximately one scheduling interval plus processing delay, even if the online query itself takes only milliseconds. This illustrates that retrieval latency and feature freshness are different measurements . Actual lag depends on source publication, scheduler delays, processing time and failures. A 15 minute cadence is not a guaranteed 15 minute freshness SLA. For 50,000 restaurants with one 100 byte serialized feature value each, the feature value payload alone is approximately 5 MB . Real store usage can be substantially higher due to keys, indexing, versions, metadata, replication and additional features. Feast commands After you have defined a feature repository and applied its configuration: The dates are illustrative. The initial and incremental materialization commands must use supported syntax for the installed Feast version. Incremental materialization tracks the previous materialization boundary and advances through configured data source time windows; it should be scheduled with cutoffs for which upstream data is actually ready. A tiny online verification example: This will work only after the feature view, entity key, source and online backend have been configured, the registry has been applied, and values have been loaded. The code does not calculate the restaurant average by itself. Why incremental materialization needs care A batch boundary such as 10:00 AM tells the next job which event time window to scan. But imagine an event recorded at 9:50 AM does not appear in the upstream table until 10:20 AM . A later incremental job starting at 10:00 AM event time may miss it. Mitigations include: Delay the materialization cutoff until a trustworthy upstream watermark indicates data readiness. Reprocess a bounded overlapping window for permitted late arrivals and corrections. Use periodic reconciliation or backfills, with event time aware conflict handling. Track source availability and processed high water marks independently when necessary. Test what the specific Feast/online store implementation does with old events and duplicate updates rather than assuming exactly once processing. Do not blindly replace newer online feature values with stale ones when replaying overlapping windows. Preserve update ordering and confirm the backend's behavior. Batch versus streaming updates Scheduled materialization is useful when source features arrive in batches or freshness budgets allow periodic refreshes. It is often simpler to operate and retry. Streaming or push ingestion may update serving features more frequently. Feast supports push based writes for appropriate sources, but this does not eliminate the need for reliable upstream processing, validation and monitoring. This lesson focuses on materialization jobs rather than streaming architecture in depth. Production safeguards Monitor the last successful materialization time, processed entity count, write failures, retry counts and source to online lag. Measure stale/missing feature values on sampled online reads. Avoid concurrent jobs overwriting one another without coordinated boundaries. Use durable scheduler state and bounded retries; reruns should be safe and observable. Validate that entity keys, feature types and expected source columns match the registered feature view. Protect against partial completion: don't report a whole refresh successful merely because some entities were updated. For backfills, avoid publishing historical values in a way that corrupts the latest online state. Keep credentials, access policies and network configuration appropriate to the online store. What materialization does not guarantee Instant freshness after an upstream event. That every historical feature version is available through the online store. A correctness guarantee for historical point in time joins (covered in Question 92). Automatic repair of late arriving data, misordered events or incorrect transformations. That every prediction request will find a non null feature. In short: Materialization loads eligible feature values into fast online storage using scheduled or incremental jobs. The hard production requirements are reliable boundaries, safe repeated writes, correct update ordering, visible freshness and verification of actual online values . Question 94 explores how freshness guarantees affect model training and inference.
How do feature freshness guarantees affect training and inference?
A feature freshness guarantee specifies how current a model's input data must be when it is used. It connects feature pipeline reliability with prediction quality: a fast lookup can return an old value, and a valid historical training example may need to simulate the same freshness rules that production applies. Question 93 explained how materialization moves values into online storage. This article explains how much staleness is acceptable, how to detect it, and how stale inputs influence training and inference . Beginner friendly example: delivery time prediction A restaurant delay feature is refreshed periodically. Its business requirement is no more than 15 minutes old at prediction time. The latest feature value is avg restaurant delay = 8 minutes. Its last verified usable event timestamp is 10:00 AM . A new prediction arrives at 10:12 AM . Feature age is 12 minutes , so it meets the 15 minute freshness requirement. At 10:19 AM , the online store still responds in 5 milliseconds, but the same feature is now 19 minutes old . It exceeds the limit by 4 minutes . The lookup is fast, yet the data is stale . The prediction service must follow an explicit fallback policy rather than silently assuming that the value remains acceptable. The feature's last event timestamp, publication timestamp, and most recent successful pipeline update can convey different information; choose the appropriate freshness signal for the feature. What does a freshness guarantee contain? Freshness limit: The maximum permitted age, such as 15 minutes for restaurant traffic or 24 hours for a slowly changing business classification. Measurement clock: Usually prediction time minus the latest valid feature event time or publication time. Document which. Scope: Per entity, feature view, store, region, or request. A healthy global pipeline does not prove every restaurant has recent values. Evaluation rule: What qualifies as stale, unavailable, or missing, including time zone and boundary semantics. Service objective: For example, at least 99% of eligible requests receive features inside their freshness budget over a rolling measurement window. Breach action: Fallback, error, alternate model, throttling, user visible degraded mode, or alerting. A freshness SLO is an internal reliability objective; an SLA is a formal service commitment, potentially with contractual consequences. Use these terms deliberately. Numerical freshness SLO example Suppose the service makes 10,000 predictions during a monitoring window and requires fresh features for 99% of eligible requests. 9,850 requests received in budget values. 150 had missing or stale values. Freshness compliance = 9,850 / 10,000 × 100 = 98.5%. The target requires at least 9,900 compliant requests. Therefore the service misses the target by 50 requests , or 0.5 percentage points . Measure this per critical feature or feature bundle, not only globally: a problematic region could disappear in an average dominated by healthy traffic. When calculating compliance, define the denominator and distinguish expected inactivity from an outage. How stale features hurt inference 1. Wrong context: A model predicts using an old traffic score after a major road closure. 2. Training serving skew: The model was trained mostly with recent features, but production frequently supplies older ones. 3. Unexpected missing values: A feature is dropped on freshness breach; the model was never trained to handle that missing state. 4. Biased behavior: Some customer groups, locations, or entities receive stale features more frequently than others. 5. Silent degradation: The online database returns HTTP 200 and low latency even while the feature computation pipeline has stopped. A freshness guarantee does not guarantee the feature value is otherwise correct; schema validity, entity mapping, transformation logic, and distribution drift remain separate concerns. How freshness affects training Historical model training data should reflect what the feature system could have supplied at each historical decision time . Assemble training rows with correct historical entity timestamps and feature snapshots. Preserve, or accurately reconstruct, the age of each input feature when the prediction would have been made. Apply the same freshness cutoff and fallback rules that the online model uses. Include missing/stale indicators or representative fallback examples if they are part of the serving contract. Evaluate model quality separately for fresh and stale/degraded cases. Avoid using today's refreshed or corrected feature values as though they were available to an older training example. For example, if production rejects restaurant delay features older than 15 minutes , but training always uses the most recent hour old value without marking it stale, offline and online behavior differ. A model can appear stronger in validation than it will be in deployment. Point in time correctness (Question 92) prevents information after a prediction cutoff from leaking backward. Freshness policy asks whether an eligible value is recent enough . These are complementary checks, not identical rules. Different features need different budgets Transaction risk signals: Could require seconds of freshness if the system makes decisions against rapidly changing activity. Current courier availability: May require seconds or minutes for effective dispatch. Restaurant delay average: Might tolerate 10–30 minutes depending on business sensitivity. Restaurant category: Might tolerate daily refreshes. These numbers are hypothetical design examples, not standard vendor SLAs. The right budgets come from experiments measuring the model's sensitivity to input age, real pipeline capabilities, and business impact. Practical Python freshness guard The function demonstrates the age threshold. A production solution must also verify the timestamp's meaning, handle clock skew, missing timestamps, late arriving data, entity specific update patterns, and its configured fallback. Do not substitute ingestion time for event time without understanding the business semantics. Safe fallback policies Fail closed: For high impact decisions, decline a prediction when a critical feature is stale and return a controlled error or manual review route. Use an approved default: For lower risk use cases, use a validated default or missing value representation; validate that the model was trained and tested on it. Use a fallback model: Deploy a separately tested model requiring fewer or more stable features. Monitor its outputs and label requests as degraded. Brief last known good grace period: Only when explicitly approved, allow a longer age window temporarily and record the reason and feature age. Never let an emergency override become an invisible permanent behavior. There is no universal fallback. A stale inventory count and a stale static product category may require very different treatment. How to monitor freshness Feature age: Current request time versus the relevant timestamp of the returned value. Freshness compliance: Percentage of eligible requests meeting the agreed threshold. Stale/missing rate: By feature view, entity segment, model, and region. Pipeline lag: Latest source event compared with latest successfully processed event. Write/ingestion delays: Whether updates reach online storage after processing. Quality outcomes: Changes in prediction errors or business metrics when inputs age. Alerting: Trigger alerts before the SLO is breached; track incident duration and recovery. Tecton's documentation explicitly describes feature freshness monitoring, upstream stale records, pipeline lag and alerting on expected freshness. Feast online retrieval stores the most recently materialized values, but the freshness guarantee still needs measurement and operational enforcement appropriate to the deployment. Costs and tradeoffs A tighter freshness budget generally increases complexity or cost: more frequent updates, streaming processing, more writes, extra infrastructure, and potentially higher operational risk. A looser budget can be cheaper but may degrade prediction quality. Determine the optimal tradeoff through controlled experiments using actual feature age distributions and model performance sensitivity. For example, refreshing every 5 minutes may produce up to 12 scheduled update opportunities per hour , versus 4 per hour at 15 minute intervals. More runs can reduce typical lag, but cannot guarantee fresh features during upstream delays or system failures. Common mistakes Calling a feature fresh because its online lookup is fast. Measuring only store wide materialization success rather than per request entity freshness. Treating an internal freshness target as a guaranteed contract. Applying the same TTL to fast changing and nearly static features. Training on perfectly fresh values while serving stale ones. Overwriting stale data with a default without measuring fallback frequency. Assuming a value's ingestion time proves when the underlying event occurred. Relying on scheduled job frequency as a guaranteed maximum feature age. In short: Define freshness in business terms, measure the actual age of features at inference , mirror freshness and fallback policies in training, and alert when compliance drops. Freshness is a property of usable model inputs, not simply of database performance or materialization schedules.
How do you define ownership and discoverability for reusable features?
Feature ownership means every reusable ML feature has an accountable team that maintains its definition and reliability. Discoverability means other teams can find the feature, understand it, evaluate whether it is safe to reuse, and request access without rebuilding it. Beginner friendly example A delivery company has three teams: Delivery ML, Fraud ML, and Restaurant Analytics. Two teams independently calculate restaurant avg delay 30d, but one includes cancelled deliveries while the other excludes them. A shared feature catalog can expose a single governed definition: Name: restaurant stats:avg delay 30d. Meaning: Average delay, in minutes, for completed restaurant orders during the preceding 30 days. Entity: restaurant id. Data type and unit: Floating point minutes. Owner: Delivery Data Platform team; escalation route through its on call service. Source and transformation: Documented source dataset and version controlled computation reference. Tags: domain=delivery, subject=restaurant, status=approved, sensitivity=internal. Consumers: ETA predictor and restaurant quality model. Access: Read permission for approved ML services; restricted changes to an authorized owning team. Lifecycle: Version/review history, change requests, deprecation notice and replacement instructions. Other teams can search for restaurant , delay , or delivery , inspect the definition, and decide whether to reuse it. Why ownership matters Without an owner, a broken feature may have nobody to contact. Suppose a source field changes from minutes to seconds: the raw number 8 becomes 480 for an eight minute delay. A responsible owner should catch this contract change, arrange a safe migration and notify affected models. Assign responsibility for: 1. Definition: What the feature represents, units, entity keys, exclusions and business meaning. 2. Quality: Type/schema checks, null behavior, monitoring and testing. 3. Operations: Source availability, incident handling, service expectations and refresh coordination. 4. Change control: Review of new versions, breaking changes and deprecation. 5. Support: How consumers request help, report issues or seek access. One feature may have a business steward and an engineering operator; both responsibilities should be explicit rather than merely listing a generic team. What makes a feature discoverable? A useful catalog entry includes: A stable, descriptive feature or feature view name. A short human readable explanation and examples of intended use . Entity keys, data type, unit, valid range and null semantics. Source, transformation repository and dependencies. Owner/team, escalation channel and current lifecycle status. Searchable domain, application and sensitivity tags. Relevant consumption instructions and permitted environments. Links to quality tests, metrics and example queries. Known limitations, privacy classification and approval process. Version history and downstream consumers where lineage tools expose them. Tags make search easier, but a tag such as approved=true is only metadata and does not grant access or enforce a security policy. Numerical example: reuse reduces duplicated effort Imagine 4 teams each spend 6 engineer hours independently discovering and rebuilding a restaurant delay feature. Duplicated work: 4 × 6 = 24 engineer hours. If a shared catalog lets each team find and assess a maintained feature in 1 hour , team level discovery/reuse takes 4 engineer hours . Illustrative avoided work: 20 engineer hours , before accounting for central platform maintenance, initial feature creation and governance costs. This is not an automatic saving: a reusable feature requires support, documentation, testing and compatibility with the consumers' needs. Organizing ownership in Feast Feast uses a feature registry to catalog applied entities, feature views, feature services and associated metadata. Feature views support descriptions and free form tags, which can hold searchable ownership/domain information. <See official Feast documentation linked below. The code illustrates a feature view definition , not a complete runnable repository; it assumes a properly configured entity and source. Apply the repository through the normal reviewed deployment process, then inspect the catalog/registry to verify the definitions are available to other teams. Tecton's Feature Services also support descriptive metadata such as owner, description, and tags, depending on platform capabilities. Ownership metadata by itself is not role based access control . Governance: who can see and change features? Distinguish three permissions: Discover metadata: Some users may need to search names and descriptions without reading sensitive values. Read feature data: Consumers need explicit authorization for the underlying offline and online values. Change definitions: Only trusted maintainers should be able to change feature schemas, sources or business semantics. Enforce real access controls in the registry, storage and serving infrastructure. Do not assume a catalog tag restricts access. For example, some feature platforms separately control online data and metadata, while underlying warehouses enforce their own offline data permissions. Sensitive features may need redacted descriptions, purpose limitations, access reviews and audit logging. Avoid publishing personal information, credentials or confidential sample values in tags or documentation. Maintenance and change management Review periodically: Confirm the owner, feature semantics and consumers are still accurate. Make changes through version control: Use reviewed pull requests and tests rather than editing production definitions ad hoc. Detect breakage: Validate types, ranges, source availability and downstream compatibility. Notify users: Announce changed definitions or deprecations before dependent models break. Deprecate safely: Mark the feature as deprecated, provide a replacement and transition window, track consumers, then retire it when safe. Handle unowned features: Assign a steward, quarantine unverified feature definitions if necessary, and establish an escalation path. Track lineage: Connect the feature to upstream sources and downstream model consumers when tooling permits. A common operational rule is one documented accountable owner per feature view, plus named support and escalation contacts . Domain ownership may be shared across teams, but accountability should remain clear. Common mistakes A short feature name with no definition, units or entity key. Tags such as team=unknown that are never maintained. Treating a searchable catalog as sufficient permission to access restricted data. Changing calculation semantics without versioning or notifying consumers. Publishing a feature that no team supports. Assuming that two similarly named features are semantically identical. Keeping obsolete features searchable as if they were actively approved. Recording ownership only in a developer's notebook rather than in a maintained catalog or repository. In short: Give reusable features a clear owner, meaningful searchable metadata, documented contract, real permissions and maintenance lifecycle . A catalog is valuable when it lets another team safely understand and reuse an existing feature instead of inventing another version. The next lesson examines sharing feature definitions across training and serving.
How can feature definitions be shared across training and serving?
Share a versioned feature contract and reusable transformation implementation between model training and live inference. Both paths must agree on the feature's meaning, inputs, units, missing value rules and output type—even when one runs on historical batches and the other handles single requests. Question 95 covered ownership and discovery. Here the engineering challenge is preventing different computations for the same feature . Worked example: delivery time prediction A model uses delay ratio, defined as restaurant delay divided by estimated travel time, clipped to a range from 0 to 5. Historical training row: restaurant delay min = 8, travel time min = 20. Shared definition: min(max(8 / 20, 0), 5) = 0.40. Live inference for the same input values must also return 0.40 , not 40 , 0.4% , or a value calculated with seconds instead of minutes. If the online implementation accidentally divides by travel time in seconds (1,200 instead of 20), it computes 8/1200 ≈ 0.0067. That is a 60× smaller value than expected. The model sees training serving skew even though the source records are otherwise correct. One shared, testable transformation For simple deterministic transformations, keep one pure function in a versioned Python package, usable from batch processing and inference services: Training: apply the function to eligible historical records before fitting the model. Serving: use the same tested package version on current eligible feature values. Package once, pin its version in both environments, and verify identical output for the same inputs. This is easy for small transformations. For very large datasets, calling a Python function row by row may be inefficient; share a common declarative definition or generate equivalent warehouse and serving implementations, then test their results against each other. How a feature store helps Feast FeatureViews define entities, data sources, fields and schema for both offline historical retrieval and online serving. OnDemandFeatureViews can define Python transformations applied in historical and online retrieval paths; their current documented support and scaling limitations must be checked for the installed Feast version. <See official Feast FeatureView documentation. A shared repository can contain: 1. The feature views and entity contracts. 2. Transformation functions or authoritative SQL expressions. 3. Feature services identifying the exact bundle expected by a model. 4. Schema and golden input tests. 5. Versioned configuration and release records. A FeatureView by itself does not guarantee that upstream transformations are identical across independently implemented batch and streaming pipelines. Validate the complete calculation path. Define the feature contract explicitly For delay ratio v1, document: Entity: Restaurant or delivery request key used for the lookup. Inputs: restaurant delay min and travel time min, both floating point minutes. Output: Float in the inclusive range 0–5 . Formula: Clamp delay / travel to the allowed range. Null handling: Do not silently replace missing values with zero; follow an agreed missing value policy. Zero denominator: Reject or use a separately designed fallback, identical in training and serving. Time meaning: Which historical feature values are eligible for a training or inference request. Version: An immutable or traceable definition identifier, such as delay ratio v1. Documenting a feature is not enough: automated contract tests must enforce it. Detailed general schema validation is addressed in Question 97. Versioning and synchronized deployment Suppose the team wants to change the formula to include weather risk. Do not silently change delay ratio v1 in place while an already trained model expects the old meaning. A safe pattern: 1. Create delay ratio v2 with a clearly described new contract. 2. Publish its implementation and run equivalence tests across historical and online paths. 3. Backfill or populate required feature values where applicable. 4. Train and evaluate a new model using v2, recording the feature definition revision. 5. Deploy compatible feature serving code and validate availability. 6. Roll out the new model only when all required v2 features can be served. 7. Keep v1 available until older model deployments and rollbacks no longer depend on it. Compatibility is a pair: A model version and the feature definition versions it expects must be deployed together or protected by backward compatible serving interfaces. A consistency test with numbers Construct a small golden dataset: Input delay=8 min, travel=20 min → expected 0.40 . Input delay=60 min, travel=10 min → expected 5.00 after clipping. Input delay=0 min, travel=10 min → expected 0.00 . Input delay=8 min, travel=0 min → expected validation failure. Run exactly those cases through the historical training transformation and the online serving transformation. Assert the same outputs, error behavior, units and data types. For floating point results, use a declared tolerance such as abs(offline online) <= 1e 6 when appropriate; do not demand bitwise equality between different numeric runtimes without reason. Also compare sampled materialized online feature values with the corresponding expected offline snapshots at compatible timestamps . An age related mismatch is different from a transformation logic mismatch. Common implementation patterns Single library: Reuse a common function across training and inference, suitable when both environments can run the same language/runtime. Shared SQL or declarative specification: Keep one source definition and compile it into supported execution engines; beware engine specific date, null and type semantics. Feature store managed transforms: Use a feature platform's supported shared transform abstraction, including Feast on demand feature views where suitable. Precompute once, serve many: Produce authoritative feature values upstream and load them into online storage, limiting the amount of transformation repeated during inference. Choose based on scale, supported languages, runtime latency, infrastructure and maintainability. Limitations and production traps Reusing the same source code does not make historical data and live feature values equally recent. Different SQL engines may interpret missing values, timestamps, integer division and numeric precision differently. A feature store definition does not guarantee data ingestion, source and transformation logic are synchronized. Changing a shared library may break previously trained models unless package versions are pinned. Request time features such as live trip distance might not exist in historical warehouses; training needs representative historical request values. Expensive Python transformations during large offline retrieval may cause unacceptable costs; use scalable execution for heavy computations. Test rollback: restoring an old model may require restoring its expected older feature interface. In short: Share and version the feature contract, transformation implementation, golden tests, and model to feature dependencies . Deploy feature definitions and trained models compatibly, then verify offline/online outputs. Reuse reduces training serving skew, but it does not replace freshness, timestamp and data quality controls.
How do you validate schemas at ML pipeline boundaries?
Schema validation checks whether data entering or leaving an ML pipeline step matches a declared contract. It catches missing columns, unexpected types, nulls, invalid ranges, unsupported categories, and incompatible model inputs before incorrect data reaches training or production inference. Question 96 covered sharing transformation definitions across training and serving. This lesson focuses on checking the actual data at each handoff , rather than defining how features are computed. Beginner friendly example: delivery time prediction A delivery model expects the following inputs: order id: required nonempty string. distance km: required numeric value, 0–100 . traffic level: required category among low, medium, high. restaurant delay min: optional numeric, 0–120 when present. event timestamp: required valid timezone aware timestamp. A newly ingested batch has 1,000 rows : 970 rows satisfy all the rules. 12 have missing distance km. 8 have distance km = 5. 10 contain an unexpected traffic level = extreme. Assuming these problems occur in distinct rows , there are 30 invalid rows . The valid row rate is 970 / 1,000 × 100 = 97%, and the invalid row rate is 3% . If the pipeline policy requires at least 99% valid rows , the batch fails , even though 970 records look usable. If violations overlap, calculate distinct invalid rows rather than summing failures. Where should you validate? 1. Ingestion boundary — source to landing area Assert expected columns and parseable data types. Reject missing keys or malformed timestamps. Flag out of range values and unexpected category codes. Quarantine invalid records with a reason; preserve original input for diagnosis if privacy rules permit. 2. Transformation boundary — raw to features Check that calculated features have correct names, types, ranges, null semantics and units. For example, distance km must not silently become meters. Validate output row counts and entity key uniqueness when required by the feature design. 3. Training boundary — features to model fitting Check required model features, feature order where applicable, consistent categorical encodings and expected target/label presence. Reject data that would silently change the model's input contract. Validate label availability and type only where labels are expected ; labels typically are not supplied by live inference requests. 4. Inference boundary — request to model Validate incoming payloads against the deployed model's signature or API schema. Return a controlled input error for missing/unsupported fields; don't let a malformed request trigger an unhandled model exception. Also verify feature store lookup outputs after retrieval. 5. Model output boundary Validate predicted response structure and type, such as a finite numeric eta minutes. Check contractual constraints and handle NaN/infinity safely. Do not confuse a structurally valid output with an accurate prediction. What counts as a schema contract? A practical schema defines: 1. Required columns and keys: Names, optional versus mandatory fields, unexpected column policy and row/entity key requirements. 2. Types: String, integer, float, timestamp, boolean or supported categorical representation. 3. Nullability: Whether a missing value is permitted and how it is represented. 4. Ranges: For example 0 <= distance km <= 100, including inclusive boundary rules. 5. Categories: Known values for traffic level, plus an explicit unknown category policy. 6. Semantic constraints: Units, timestamps, uniqueness and cross field rules. 7. Compatibility: Which producer schema versions a pipeline stage or deployed model accepts. Schema checks differ from statistical monitoring: A dataset could pass every type/range check while its distribution changes drastically. Drift detection and model quality monitoring are separate checks. A simple executable Python validator This intentionally small example covers a subset of the full contract; it does not validate event timestamp, unexpected keys, provenance, or batch uniqueness. Production pipelines should use an appropriate validation library or structured schema implementation, covered by tests. Validating a whole dataset with Great Expectations Great Expectations (GX) supports schema expectations such as required column existence, column data types, ordered column lists, acceptable value sets and numerical ranges. Conceptually, an ingestion checkpoint might enforce: The GX documentation includes ExpectColumnToExist, ExpectColumnValuesToBeOfType, ExpectTableColumnsToMatchSet and range/category expectations. Use versions and APIs supported by your environment; validate the source dataset through a committed, reviewed expectation suite. TensorFlow Data Validation (TFDV) also detects anomalies by checking dataset statistics against an explicit schema. Its training/serving schema environments let a label column be required for training but absent in serving without raising a false error. Backward compatibility decisions A producer adding an optional field can be safe if consumers tolerate unknown columns . Removing a required field, changing distance km from float to string, or changing units from kilometers to meters can break consumers. Compatible addition: New nullable column that old consumers ignore. Potentially breaking change: Renaming an input column, altering its type/units, or making an optional field required. Category extension: A new traffic level category may be structurally representable yet unsupported by the model's encoder. Model specific contract: The same source schema can feed different models with different required subsets of columns. Always test compatibility against actual downstream versions; do not declare a change safe solely because it is additive. Question 99 covers schema evolution in greater detail. What happens when validation fails? Fail closed for critical fields: Stop training or reject the inference request when required inputs cannot be trusted. Quarantine records: Isolate invalid data with concise reason codes and auditable batch IDs. Apply an approved fallback: Only for explicitly optional fields and models tested with those fallback values. Notify the owner: Alert the upstream producer or pipeline owner with counts, failure categories and sample references that do not expose sensitive data. Retry appropriately: Retry transient read/service failures, not deterministic schema incompatibilities. Prevent partial publication: Do not mark a dataset or artifact validated if critical checks fail. Do not automatically coerce everything: blindly converting malformed strings or out of range values may hide upstream defects. Production test checklist Run golden valid and intentionally invalid datasets through every relevant boundary. Test missing columns, extra columns, unexpected categories, nullable fields, NaN/infinity, units and timestamp parsing. Verify label required training schemas are distinguished from label free serving schemas. Assert reject, quarantine, alert and fallback behaviors. Track validation failure rate, affected entities, schema version, release ID and top failure types. Recheck the model's actual serving signature before deployment. Version the schema with the pipeline code and pin the schema accepted by each model. In short: Put explicit, versioned schema checks at ingestion, feature transformation, training, inference and output boundaries. Reject unsafe incompatibilities early, record clear failures, and test compatibility for every pipeline consumer.
What is a data contract between upstream sources and an ML pipeline?
A data contract is a versioned, testable agreement between an upstream data producer and the ML team consuming its data. It specifies not only which columns exist , but also what values mean , when data should arrive , how quality is measured , who is accountable , and how changes are introduced safely . Question 97 explained validating schemas at pipeline boundaries . A data contract goes further: it defines the producer–consumer operating agreement that those checks help enforce. Question 99 will cover detailed schema evolution strategies. Practical example: food delivery ETA model The Orders Platform publishes an orders completed dataset. The ETA ML Team consumes it to train a delivery duration prediction model. They agree to contract orders completed v1: Producer: Orders Platform; responsible for publishing and correcting the dataset. Consumer: ETA ML Team; responsible for training transformations and reporting contract failures. Required fields: order id (nonempty string), restaurant id (string), delivered at (UTC timestamp), delivery duration min (numeric). Semantics: delivery duration min means minutes from courier pickup to customer handoff, not order placement to delivery. Cancelled orders are excluded. Quality: Unique order IDs, no nulls in required fields, and nonnegative duration under an agreed upper bound. Delivery: Completed order data available in the agreed warehouse table by 06:00 UTC daily for the previous day's data. Availability SLO: At least 99% of scheduled daily deliveries meet that deadline, measured over the contract's defined assessment window. Support and escalation: Producer on call and named ML consumer; incident notifications through the agreed channel. Retention and access: Documented storage lifetime, permitted uses, sensitive data classification and access controls. Versioning: Contract version and source location stored with downstream training runs. The example is fictional. These are negotiated requirements, not universal service levels. Why schema validation alone is insufficient Suppose upstream delivery duration min continues to be a float, but the producer silently changes its meaning from courier pickup to handoff to order placement to handoff . An order that spent 12 minutes waiting before pickup and 18 minutes in transit was previously recorded as 18 minutes . After the definition change, it becomes 30 minutes . Both values are positive floats, so a type/range test may pass. Yet the target has changed by 12 minutes , a 66.7% increase relative to 18 minutes, and historical training labels are no longer comparable. The contract's semantic definition and change approval process catch what a structural schema test cannot. What should a producer–consumer contract contain? 1. Structure and schema Dataset/table/topic identity, keys, column names and data types. Nullability, units, allowed categories, timestamp conventions and whether unknown columns are permitted. Version of the contract schema accepted by each consumer. 2. Business semantics Precise meanings, inclusion and exclusion rules, aggregation windows and label definitions. For ML, whether attributes are available at the intended prediction time, and documented known limitations. Definition changes that could alter model behavior even if the data type is unchanged. 3. Data quality expectations Uniqueness, completeness, validity, referential integrity and domain specific constraints. Explicit thresholds, sampling rules, severity, and actions when a rule fails. Reports and evidence from automated tests. 4. Availability and service levels Publication schedule, delivery deadline, expected update frequency, retention period and recovery objectives. Measurable SLO/SLA definitions and an escalation path for outages. How late deliveries, corrections and replays are communicated. 5. Ownership, governance and usage Producer and consumer contacts, approvers, incident responsibilities. Data classification, permitted purposes, access and auditing requirements. How to discover the authoritative contract, obtain credentials securely and request exceptions. 6. Versioning and changes Contract identifier, version, effective date and revision history. Consumer review and notification before incompatible changes. Deprecation window and tested migration path when the agreement must change. Numerical delivery SLO example Suppose the agreed SLO is 99% on time delivery of daily snapshots over a 100 day measurement period. Expected snapshots: 100 . Received by 06:00 UTC: 97 . Received late: 3 . On time delivery rate = 97 / 100 × 100 = 97%, which fails the 99% objective . The ML team must know what to do when data is late: defer training, reuse a previously validated snapshot if that is explicitly permitted, or escalate. A late input should not be silently presented as up to date. A readable illustrative contract This is conceptual YAML , not guaranteed to conform to the Open Data Contract Standard or any vendor's accepted syntax. Real implementations should follow a chosen standard's schema, validate that contract document itself, and attach machine executable dataset checks. The Open Data Contract Standard (ODCS) is one published format describing schema, data quality, service levels, teams and support channels. OpenMetadata also supports data contract concepts for governance and quality evaluation. How contract testing works 1. Producer CI: When a data source or transformation changes, validate its output against the contract and run representative semantic tests. 2. Pre publication checks: Confirm schema, quality thresholds and required metadata for the output snapshot. 3. Consumer CI: Test the candidate producer output against the versions accepted by deployed ML pipelines; run contract tests using fixtures that cover known edge cases. 4. Runtime checks: Confirm each delivered batch has the expected contract version, availability timestamp and quality result. 5. Evidence and audit: Persist the validation result, tested dataset identity, contract revision, and responsible service. Tests should block incompatible production changes before consumers are switched where feasible. Runtime monitoring still matters because source outages and data anomalies can occur after CI passes. What if the producer breaks the contract? For example, tomorrow's data arrives with duration sec instead of the required delivery duration min. Mark the incoming dataset or contract evaluation failed . Quarantine the incompatible snapshot or prevent it from becoming the approved training input. Notify the Orders Platform owner with the failed contract version and evidence. Continue using only a permitted prior snapshot, or postpone model training. Restore compliant data or agree on a reviewed contract migration before accepting the change. Record the incident and downstream impact. Do not automatically convert seconds into minutes unless that compatibility adaptation has been reviewed, tested and documented; accidental coercion can hide larger semantic changes. Change notifications and safe evolution Potentially compatible: Add an optional, documented field that existing consumers ignore; still test the actual consumers. Potentially breaking: Rename a required field, change its type/units, or modify a feature or label's definition. Notify before cutover: Share the proposed change, examples, expected timing, downstream impacts and a test dataset. Enforce approval: Require impacted consumer sign off where policy demands it; retain the earlier interface until consumers are ready. A useful policy is producer proposes → consumers validate → both approve → rollout is monitored . Question 99 examines the mechanics of evolving schemas in more detail. Limits and tradeoffs A contract cannot prevent all statistical drift or guarantee that a trained model remains accurate. Semantic tests may require domain knowledge and cannot always be reduced to column checks. Negotiating and maintaining agreements costs time; start with the most business critical datasets. An SLO is meaningful only if it has a monitoring definition and an incident response process. Contract documents are not enforcement by themselves: integrate tests, ownership and change gates into real publishing workflows. In short: Treat an upstream dataset like a versioned service interface . Define its structure, meaning, quality, delivery commitments and owners; test both producer and consumer behavior; and require communication and approval for breaking changes.
How do you handle schema evolution without breaking a training pipeline?
Schema evolution means changing the structure or interpretation of upstream data while keeping existing training jobs and model serving consumers working. The safest approach is to make compatible changes first, version incompatible changes explicitly, test old and new consumers, then roll out gradually with a rollback path. Question 98 covered the producer–consumer data contract itself. This lesson focuses on how to migrate an existing schema without interrupting training . Beginner friendly example A food delivery training pipeline expects: order id: string. distance km: float, measured in kilometers . delivery duration min: float, measured in minutes . The Orders team wants to introduce a useful feature, weather code, and later rename distance km to distance meters. Change A — add an optional field: Adding nullable weather code can be backward compatible for old consumers if they ignore extra fields . New consumers should define how to handle historical rows where this feature is missing. Change B — rename and change units: Replacing distance km with distance meters is potentially breaking. A legacy training step expecting distance km will fail or, worse, consume meters as kilometers if the name or semantics are confused. For a 7.5 km delivery: Old representation: distance km = 7.5. New representation: distance meters = 7,500. Without conversion, an old model expecting kilometers could see an input 1,000 times larger . Correct adapter: distance km = distance meters / 1000 = 7.5. Notice the risk: a syntactically valid float is not proof that units and meanings are compatible. Backward versus forward compatibility For schemas used by a producer and a consumer: Backward compatibility: A reader using the new schema can consume data written with the old schema . Forward compatibility: A reader using the old schema can consume data written with the new schema . Full compatibility: Both directions are supported. Transitive compatibility: The selected rule is checked against all supported previous schema versions , not just the most recent one. These definitions match Confluent Schema Registry documentation. Actual allowable changes depend on serialization format (Avro, Protobuf, JSON Schema), defaults, reader implementation and pipeline code. For example, adding a nullable/defaulted field is often safe in Avro; adding a required field with no valid default can break readers of older records. For ML pipelines, compatibility also involves business meaning, model expectations and units —which basic serialization schema compatibility checks may miss. Migration pattern: expand → migrate → contract 1. Expand: Introduce a new optional field, or publish both old and new fields during a transition. Keep existing readers functional. 2. Adapt: Add a tested conversion in the ingestion or feature transformation boundary. Record which input schema version was used. 3. Migrate: Upgrade training pipelines and deployed models to consume the new canonical representation. 4. Verify: Compare values and retraining results on representative old and new data. 5. Contract: Remove the old field only after every supported consumer has migrated and old dataset replays have a valid compatibility path. For example, publish both distance km and distance meters temporarily. Validate that abs(distance meters / 1000 distance km) is within an approved numerical tolerance, and alert on mismatches. Avoid permanent dual write complexity: define a deprecation deadline and keep a plan for older stored data. Simple Python adapter This illustrates conversion, not a complete production adapter. If both fields are present, real code should check that they agree instead of automatically trusting one. Also reject booleans and invalid type coercions according to your data contract. Defaults: useful, but not magic Suppose the new training pipeline expects weather code, but six months of historical records have no weather value. A nullable field and explicit unknown category may be appropriate if the model is trained to handle them. Filling every missing value with sunny would invent weather observations and may distort training. An Avro field default helps schema resolution; it does not prove that substituting the default is statistically safe for the ML model. For retraining with historical data, a backfill from a trustworthy historical weather source may be better, if available. Document when defaults are used and test model performance on those cases. Compatibility tests before deployment Producer tests New records satisfy their declared schema and required semantic constraints. Old fields remain available during the compatibility period. Conversion between units is accurate across boundary and typical cases. No unsupported null/category values are introduced. Consumer tests The current production training job still reads new producer output. The new training job reads both old stored records and new records. Model feature order, dtype and units remain what the trained model expects. An expected failure blocks promotion and produces a clear diagnostic. Historical replay Load representative snapshots from previous schema versions. Check the normalized features for identical business meaning. Train or dry run a small pipeline on old and new snapshots; compare feature statistics and metrics. Preserve version IDs and transformation code revisions to explain any difference. A staged rollout example Suppose there are 10 downstream training pipelines depending on distance km. 1. Begin dual field publication without removing distance km. 2. Upgrade 2 pipelines to the v2 adapter and run them with shadow validation. 3. If results match, upgrade the remaining 8 pipelines in controlled batches. 4. Confirm that 10/10 registered consumers use compatible logic and historical replays work. 5. Only after a documented retirement window, stop emitting the old field. If the first two pipelines show unexpected feature changes, revert the consumer adapter while keeping the old producer field available. A successful rollout is not measured only by whether code deploys—it must preserve the model input contract. Rollback planning Before rollout: Save the old schema definition, producer deployment, adapter version and approved dataset snapshots. During rollout: Keep a dual field or backward readable path for older consumers. On failure: Pause the cutover, restore compatible producer or consumer configuration, and rerun affected pipeline jobs using trusted snapshots. After rollback: Verify that ingestion, transformation, training and model predictions receive the intended values. Long term: Do not drop old schema support until data retention and replay requirements no longer depend on it. A schema registry may reject incompatible wire format changes, but it cannot automatically roll back producer semantics or retrain a model. Common mistakes Renaming a field without updating historical datasets and downstream consumers. Changing kilometers to meters while keeping the same feature name. Treating nullable/defaulted fields as automatically harmless for model quality. Deploying a new producer before checking supported old readers. Testing only the newest schema instead of replaying older snapshots. Keeping two conflicting definitions indefinitely without ownership. Deleting deprecated columns before confirming all consumers have moved. Assuming a BACKWARD setting checks every historical schema version; Confluent's default BACKWARD is non transitive unless explicitly configured otherwise. In short: Evolve schemas with versioned changes, compatibility checks, explicit unit/semantic adapters, gradual consumer upgrades, historical replay tests and a reversible rollout . Do not confuse structural compatibility with ML feature correctness.
How do you quarantine invalid records instead of silently dropping them?
Quarantining invalid records means preserving them in a separate, controlled location—with an explanation of why each record failed—instead of silently deleting them from an ML dataset. Valid records can continue through the pipeline when policy allows, while invalid records remain available for diagnosis, correction and safe replay. Question 99 addressed schema evolution . This lesson focuses on what to do with individual bad records after validation , rather than changing the schema or dropping evidence of a failure. Beginner friendly example: food delivery data An ingestion job receives 10,000 delivery orders for training a delivery time prediction model. It checks: order id must be present and unique. distance km must be a finite number between 0 and 100 . delivery duration min must be nonnegative and at most 240 . delivered at must be a valid timestamp. Validation finds 9,700 valid records and 300 distinct invalid records : 120 orders have negative distance. 100 have a missing order ID. 80 have malformed timestamps. Assuming these error groups do not overlap: Valid rate: 9,700 / 10,000 × 100 = 97%. Quarantine rate: 300 / 10,000 × 100 = 3%. Without quarantine: The pipeline might quietly train on 9,700 rows and lose the ability to explain why 300 orders disappeared. With quarantine: It writes 9,700 valid rows to the approved dataset and saves 300 failing rows with their original values and reason codes. The model training pipeline can proceed only if the dataset's quality policy permits a 3% rejection rate . A safety critical or high bias dataset may need to fail the entire batch instead. Typical quarantine flow The key is to preserve enough evidence to understand the failure without exposing protected data unnecessarily. What should each quarantined record contain? Original record or a protected pointer to the immutable raw event. Source location and batch/run ID so the record can be traced back to ingestion. Stable row/event identity , such as a source event ID or source partition and offset. Failed checks , for example NEGATIVE DISTANCE or MISSING ORDER ID. Validation rule version and relevant schema/contract revision. Detection timestamp and processing job identifier. Status , such as NEW, UNDER REVIEW, FIXED, REPLAYED or EXEMPTED. Repair metadata: Who changed it, why, when, and which replacement record was produced. Keep the original value intact for audit. Store corrected data as a separate revision rather than silently overwriting the evidence. Small Python demonstration This example demonstrates the split, not durable production storage . A real job must validate all required fields, handle duplicate IDs across records, protect sensitive values, and persist both outputs reliably. Why a dead letter or quarantine destination matters A quarantine dataset or dead letter queue makes failures observable and recoverable . Depending on the source, it might be: A restricted object storage prefix containing immutable rejected batches. A database table with one row per rejected event and reason codes. A dead letter topic/queue for events that cannot be safely processed, plus durable diagnostic storage. A message broker's dead letter queue is not by itself a complete analytics quarantine system: preserve consumer offsets, replay identity, access controls and retention as needed. Avoid data loss during partial writes Suppose a job reads 10,000 source events and writes the valid and invalid groups separately. If the valid write succeeds but quarantine storage fails, silently acknowledging the input could lose the 300 rejected events. Use a transactional or staged publish process where available: 1. Store the raw input durably before transformation. 2. Write and verify valid and quarantined outputs. 3. Record batch counts and lineage for both destinations. 4. Commit the batch or advance input checkpoints only after required writes succeed . 5. Make retries idempotent using stable record keys and batch IDs. For nontransactional storage, use manifests and explicit completion markers to prevent an incomplete batch from looking approved. Repair and replay safely Example: 120 rows have distance km = 4 due to a known upstream sign conversion bug. 1. Identify and document the verified root cause. 2. Produce a corrected copy of the affected rows, preserving the originals. 3. Re run the current approved validation rules on corrected records. 4. Confirm uniqueness and check that the corrected events have not already been processed. 5. Replay only the approved replacements, using idempotency keys. 6. Update quarantine status and link to the replay outcome. Never automatically replace 4 with 4 just because the absolute value seems plausible. The original value may indicate a different upstream defect. Corrections should be supported by a trustworthy source or explicit business rules. Monitor quarantines, not only pipeline success Quarantine count and rate: Number and percentage of distinct rejected records by batch. Error categories: Most frequent failed rules and upstream sources. Recovery time: Time from detection to repair or explicit disposition. Replay success rate: How many corrected records passed revalidation and were committed exactly once at the logical processing level. Impact on training: Whether rejected records are concentrated in a geographic region, customer segment or rare class. Volume change: A sudden rise in rejected records can indicate a newly broken producer. For example, a dataset might show only 3% rejected overall , but 25% of one small region's orders are missing. Proceeding without investigation could create biased training coverage. Enforce both overall and important segment level policies. Tools and practical implementation choices Great Expectations (GX) reports failing validations and supports retrieving all unexpected rows for its UnexpectedRowsExpectation using ValidationDefinition.get unexpected rows(). That is useful for building a quarantine dataset; the API has expectation type limitations, so do not assume every validation automatically returns every failing row. AWS Glue Data Quality can produce row level rule results in supported Glue ETL jobs, including fields identifying which rules failed. Such results can be used to route rejected rows into an S3 quarantine location or another controlled destination. Regardless of the tool, quarantine, repair workflow and replay orchestration need to be implemented and governed explicitly. Common mistakes Dropping records without retaining failure reasons or provenance. Letting a partial success batch appear fully valid. Storing sensitive raw records in a publicly accessible quarantine bucket. Allowing an unbounded quarantine backlog without ownership or retention limits. Replaying repaired events without deduplication. Silently coercing invalid values into plausible but wrong data. Assuming a high overall valid rate means every important segment is represented. Retaining only a small sample of bad rows when complete recovery is required. In short: Treat rejected data as a tracked operational outcome , not garbage. Save the evidence securely, monitor rejection rates, investigate causes, revalidate corrections, and replay them safely—while ensuring training proceeds only when the remaining dataset satisfies policy. Question 101 examines automated dataset quality checks in ingestion jobs.