How do you evolve production ML inference API and prediction contracts without breaking clients or rollback compatibility?

Safe inference contract evolution means clients, prediction producers, feature services, and rollback releases continue to understand one another during a model upgrade—even when old and new versions run concurrently. A response that validates against JSON Schema may still be *semantically incompatible*: a score may change units, class ordering, threshold meaning, freshness expectations, or decision authority without changing its field name.

Curriculum boundary: Questions 97–99 cover schemas and evolution in *training/data pipelines*. Questions 120–125 cover reproducible packaging, validation, rollout and delivery; Question 124 covers model A/B testing; Question 135 covers production decision provenance; and Question 136 covers safe deprecation. Question 137 focuses on the live API and prediction-consumer contract during model upgrades: compatibility direction, numerical/business semantics, expand–migrate–contract, cross-version fixtures, consumer rollout order, asynchronous events, rollback and safe removal of old fields. It does not repeat generic schema validation or model retirement.

Beginner-friendly production incident

A delivery-risk service used to respond:

JSON
{"risk_score":0.82,"risk_label":"HIGH","model_version":"r27"}

Its new model, r28, returns:

JSON
{"risk_score":82,"risk_label":"HIGH","model_version":"r28"}

Both outputs include the same three fields and may satisfy a permissive schema where risk_score is merely number. But r27's score is a fraction in [0,1], while r28's score is a percentage in [0,100]. A client that sends orders to manual review whenever risk_score >= 0.75 now reviews nearly every r28-scored order.

The breaking change is semantic, not structural. The correct contract specifies score definition, range, units, class mapping, model-specific calibration and business-decision threshold policy. If the new model genuinely produces different probabilities, merely retaining the [0,1] scale is not proof of equivalent decisions; validation must include the resulting business actions.

Six layers of an inference contract

• Request syntax: Path/version, HTTP method, authentication, content type, required fields, types, defaults and maximum payload size.
• Input meaning: Units, timestamp time zone, missing-feature semantics, categorical vocabulary, numerical ranges, entity/tenant identity and required freshness.
• Output syntax: Field names, types, enumerations, nullable values, error response shapes, correlation IDs and model/version provenance.
• Output meaning: Score interpretation, probability calibration, label order, positive class, threshold/policy revision, confidence semantics and fallbacks.
• Delivery semantics: Synchronous result versus async job, completion guarantee, retried requests, idempotency, event ordering and snapshot/version pinning.
• Operational promise: Response deadline, approved degraded behavior, retained old-version compatibility, client notification and rollback window.

Treat each as independently versioned where appropriate. A stable /v1 endpoint can use different model revisions internally only if the documented consumer contract remains valid.

Backward, forward and full compatibility: direction matters

For producer/consumer data schema evolution, define:

• Backward compatibility: A new consumer can read data emitted by an older producer.
• Forward compatibility: An old consumer can read data emitted by a new producer.
• Full compatibility: Both relationships hold.
• Transitive compatibility: The proposed schema is checked against all supported earlier versions, not merely the immediate predecessor.

Confluent Schema Registry documents these definitions and notes that its default BACKWARD compatibility mode is nontransitive. If new producers must coexist with old consumers in an inference event pipeline, you need the old-consumer/new-producer direction (forward compatibility), or full compatibility, plus application-level semantic checks. Compatibility depends on format and parsing rules: optional fields, defaults, enum evolution and unknown-field handling differ between Avro, Protobuf, JSON Schema and concrete HTTP clients.

Critical distinction: Schema Registry compatibility does not prove ML decision equivalence, probability calibration, fairness, or that a client tolerates a changed enum. Test actual clients and business behaviors as well as schemas.

Numerical example 1: one unit error, thousands of wrong decisions

At 200 predictions/sec, an endpoint begins a 10% candidate rollout lasting 15 minutes.

• Eligible traffic in the window: 200 × 15 × 60 = 180,000 requests.
• If the realized candidate share is exactly 10%, the new version serves 18,000 requests.
• Suppose 12% of those candidate results cause a wrong client decision due to the score-unit mismatch. Impacted decisions = 18,000 × 0.12 = 2,160.
• Overall impacted share = 2,160 / 180,000 = 1.2%.

This is an illustrative assumption, not a predicted error rate. The canary may appear to have low HTTP error rate and p95 latency while producing a serious semantic regression. Gate on authorized business decisions and version-specific output interpretation, not just 200s and JSON validity. Count actual per-release requests rather than inferring exact shares from configured traffic weights.

Numerical example 2: nulls, defaults and consumer compatibility

Assume v1 response contract has risk_band with values LOW|MEDIUM|HIGH; a mobile client has an exhaustive switch that crashes on unknown labels. The proposed v2 adds UNKNOWN for missing features.

• Adding an enum value is often a semantic breaking change for older strict clients even when a schema checker accepts it.
• With 50,000 daily predictions and 2% of responses now tagged UNKNOWN, 1,000/day reach the new branch.
• If 70% of those go to older client versions that crash, there are 700 affected responses/day despite the server succeeding.
• Safer options: emit the previously specified error/deferred result under the v1 contract, make a new API contract version for UNKNOWN, or upgrade old clients first and explicitly verify unknown-enum handling.

Never reinterpret null as zero probability without business approval. Null, absent, stale, invalid and legitimately low risk are distinct states.

Numerical example 3: transitive compatibility gap

Suppose a publisher emits contract v1 with eta_minutes; v2 introduces an alias eta_seconds and keeps the old field; v3 removes eta_minutes.

A v3 schema might be judged compatible with v2 under a particular configured rule, but a v1 client still expects eta_minutes and fails. The break occurs when v3 is compared with all still-supported client versions, not only v2.

If the organization supports v1 and v2 clients simultaneously, use a transitive compatibility policy that matches the actual consumer/producer direction, plus end-to-end consumer tests. The simple example is conceptual; whether the schema registry accepts these exact fields under a chosen mode depends on optionality/defaults and the encoding format. A green registry check alone is not evidence that v1 clients are safe.

Production workflow: expand → migrate → contract

Phase 1: Inventory consumers and freeze behavior

• Identify clients by team, app version, region, dependency, business action and last-seen schema version. Include internal batch jobs, streaming processors, shadow comparators and rollback services.
• Record current request/response examples and *semantic* invariants: units, output class order, freshness, error semantics and thresholds.
• Pin immutable model/image/feature-contract digests for both current and proposed releases.

Phase 2: Expand without breaking old clients

• Add optional, documented response fields rather than removing or repurposing old fields.
• Keep old field meanings identical. If necessary, serve both risk_probability in [0,1] and a *new* risk_percent in [0,100], with validated deterministic conversion where justified.
• Accept new optional request fields with safe defaults where appropriate; never silently change the meaning of an old field.
• Introduce a separate /v2 contract or content negotiation for irreconcilable changes, while maintaining a functional /v1 adapter for supported clients.

Phase 3: Upgrade consumers deliberately

• Publish versioned contract docs, OpenAPI/JSON Schema/Protobuf definitions and test fixtures.
• Update mobile/web/business rules to understand the new field, explicit missing-data states and the associated model/policy revision.
• Run contract tests on *every supported old client* and the new client, against both old and new serving versions where rollback requires it.
• Monitor actual usage of old fields, routes and client versions with privacy-safe telemetry.

Phase 4: Canary the production interface

• Keep stable and candidate releases separately attributable by actual model/contract version.
• Compare schema errors, rejected/invalid decisions, fallback frequency, p95/p99, feature freshness and business outcomes for comparable traffic.
• Observe error *and semantic* safety guardrails; do not blindly infer production equivalence from a shadow run because shadow results may not generate real business actions.
• Halt or roll back through the established controlled release process when a critical invariant fails.

Phase 5: Contract only after evidence

• Stop new use of deprecated fields; give each consumer an accountable migration owner and deadline.
• Remove fields/endpoints only when the support contract allows it, consumer inventories and real traffic confirm migration, slow batch/event readers are accounted for, and rollback no longer requires old compatibility.
• Follow Question 136 for model and dependent artifact retirement; deleting a field is an API-contract decision, not proof old model weights may be deleted.

The rollback compatibility trap

Consider r27 expecting feature input distance_km while r28 is deployed with a feature service that now emits only distance_m.

Rolling back only model weights to r27 after a canary failure does not restore compatibility. The old model can interpret meters as kilometers, or fail schema validation. A correct rollback plan preserves a coherent bundle:

• Model weights and preprocessing version.
• Input feature names, units, defaults and freshness policy.
• Output contract, response adapter and business threshold revision.
• Compatible client or event consumer behavior.
• Verified model and feature digests for the actual runtime.

Test both forward rollout and backward recovery in staging. Where a storage/feature migration is irreversible, use a compatibility adapter or a distinct recovery strategy rather than promising an impossible instant rollback.

Practical illustrative contract manifest

YAML
service: delivery-risk
api_contract: risk-predict/v1
model_release: r28
request:
  distance_km: {type: number, min: 0}
  route_id: {type: string, required: true}
  timestamp: {format: rfc3339, timezone: UTC}
response:
  risk_probability: {type: number, min: 0, max: 1}
  risk_label: {enum: [LOW, MEDIUM, HIGH]}
  model_release: {type: string}
  decision_policy_version: {type: string}
semantics:
  positive_class: late_delivery
  invalid_required_feature: explicit_error
  no_unapproved_default_probability: true
compatibility:
  clients_supported: [mobile_v1, web_v1, internal_batch_v1]
  old_client_new_release_tests: required
  new_client_old_release_tests: required
  reverse_rollout_test: required
  removal_requires_zero_remaining_references: true

This is illustrative policy YAML, not a valid OpenAPI or schema-registry configuration. Real schemas must be separately authored for each format and their validators configured; unit and class semantics need custom executable tests.

What contract tests should assert

Structural API tests

• Exact documented required fields, scalar ranges, nullability and enum behavior.
• Unknown-field behavior: whether clients ignore, reject or preserve new fields.
• Request sizes, error codes, correlation IDs, authentication and tenant scoping.
• Old client parsing of new responses and new client parsing of retained old responses.

ML semantic tests

• Output probability and numerical range, positive-class mapping, label ordering and unit conversion.
• Decision policy thresholds and abstain/defer semantics.
• Calibration/quality evaluation on approved representative data with proper label maturity.
• Missing/stale feature handling cannot silently create an apparently valid prediction.

Integration tests

• Candidate + stable mixed fleet behind same gateway and sticky-session variants.
• Old async requests executed after deployment; job model/config pinned at submission or explicitly authorized migration.
• Stream replay and older event schemas, with deduplication and schema compatibility verified.
• Batch jobs reading historical input partitions and emitting consumer-compatible output manifests.
• Fallback, cache and rollback paths that return the documented contract, not a hidden alternate schema.

Common production failure scenarios and root-cause debugging

1. Percentage score slips into a fractional field

• Symptoms: HTTP 200 and valid JSON, but client decision rate jumps sharply.
• Debug: Compare output ranges by release, response-contract ID, actual client threshold and source model calibration.
• Fix: Restore the old semantics, introduce an explicit new field/version, add threshold-level contract fixtures and decision-rate guardrails.

2. New enum breaks an older mobile client

• Symptoms: Old app versions crash on UNKNOWN, while backend metrics look healthy.
• Debug: Inspect client-version-specific failures, enum parsers and actual candidate response mix.
• Fix: Serve a compatible v1 representation, support new enum only in v2 or upgrade clients first; test every supported legacy parser.

3. Rollback restores model but not feature units

• Symptoms: After rollback the old model predicts extreme values or errors.
• Debug: Trace actual model digest, preprocessing/feature-contract revision, source units and cached representations.
• Fix: Restore a coherent release bundle or use a verified adapter; keep a feature compatibility window and rollback test.

4. A monthly batch consumer breaks after field deletion

• Symptoms: Immediate online traffic is normal; the first business-day batch run fails.
• Debug: Search scheduler manifests, schema consumer dependencies, last successful run and contract-version references.
• Fix: Re-enable compatible output or migrate consumer with a full representative run before removal.

5. Async job runs with a different model than the requester assumed

• Symptoms: A request accepted under r27 is processed after upgrade using r28, changing score semantics or threshold behavior.
• Debug: Check job metadata, queue retry behavior and whether the worker resolves a floating alias at execution.
• Fix: Pin approved model/contract/policy at submission or define and log explicit migration consent. Make idempotency/traceability consistent.

6. Contract registry passes but users receive wrong labels

• Symptoms: Schema is valid but index 0 and index 1 class interpretations were swapped.
• Debug: Compare trained model class order, serialized label mapping, client display logic and model/policy metadata.
• Fix: Version class ordering, use named output classes and add golden predictions checking *meaning* rather than only field types.

7. Old client is not in telemetry

• Symptoms: A release is declared safe to contract because all visible clients use v2, but a partner still calls v1 via a low-volume route.
• Debug: Compare edge access counts, gateway routes, partner inventory, and long-running job schedules against client telemetry.
• Fix: Treat absent instrumentation as unknown, not zero; establish explicit partner acknowledgement and expiration dates.

Monitoring and rollout verification

• Structural health: Request-schema errors, output-contract rejects, deserialize failures, unknown-enum/field errors and error responses by client/contract version.
• Semantic health: Invalid probabilities, units, model class order, confidence/fallback mix, decision-rate shifts and policy-threshold violations per version and important cohort.
• Traffic and ownership: Actual client versions, active contracts, route/request counts and last seen by supported integration (with privacy-safe IDs).
• Mixed-fleet parity: Model digest and contract ID in each response, real rollout weights, sticky routes and rollback/recovery health.
• Data consistency: Feature-contract changes, timestamp freshness, default-value frequency and migration adapter error counts.
• Business consequences: Orders escalated, false positives/negatives when labels mature, appeals and downstream process mismatches.
• Deprecation safety: Legacy field reads, old event consumers, pending async jobs, batch schedules and defined sunset acknowledgements.

Use a small set of hard safety gates (e.g., zero unauthorized decisions, zero invalid score ranges, no prohibited stale features) alongside statistical/operational performance gates. A passing compatibility test on synthetic data is not a guarantee of population-level model quality.

Engineering trade-offs

• Single stable endpoint with adapters: Simplifies clients but increases server compatibility code and may conceal many supported behaviors; test by explicit contract ID.
• Explicit /v1 and /v2 endpoints: Clear boundaries and retirement controls, but duplicated routes and longer migration cost.
• Additive response fields: Usually safer for tolerant clients, but increases payload size and does not help strict parsers or changed business semantics.
• Full/transitive compatibility: Protects more historical versions but constrains rapid schema changes; these modes are schema-format specific and do not prove numerical meaning.
• Version pinning for async work: Reproducible but requires retaining older bundles while jobs/replays are outstanding.
• Longer migration periods: Fewer broken consumers but more maintenance, security and testing overhead.
• Hard cutover: Easier eventual cleanup, but unacceptable when unowned clients, DR paths or data histories remain.
• Consumer-driven contract testing: Excellent at exposing observed dependencies, but only as complete as the consumer inventory and assertions supplied.

Common production mistakes

• Equating JSON Schema validity with correct model-score meaning.
• Reusing a field name after changing its units, classes or threshold semantics.
• Adding an enum member without testing older clients' exhaustive switches.
• Assuming BACKWARD means old consumers can read all new events.
• Relying on nontransitive validation while still supporting much older consumers.
• Upgrading a feature service irreversibly and promising simple model-only rollback.
• Removing fields after seven days of online zero traffic while monthly jobs still depend on them.
• Using mutable aliases for async jobs that must preserve submission-time behavior.
• Counting gateway 200s while client-side decisions become incorrect.
• Treating missing client telemetry as evidence that nobody uses an old contract.
• Testing stable/new model releases individually but not their mixed-version routing.
• Conflating retiring the old model with safely removing all of its API schemas.

Production checklist

• Document request, response, data-unit and decision semantics as explicit versioned contracts.
• Identify all supported online, async, batch, stream, partner, fallback and DR consumers.
• Decide compatibility direction and history depth, with correct schema-format rules.
• Introduce additive changes or a new API version rather than silently repurposing an old field.
• Exercise old and new clients against stable and candidate releases, including a real rollback.
• Validate probability/class/threshold meanings and feature freshness with golden fixtures.
• Monitor schema, semantic and business decision outcomes per client and model release.
• Only remove old fields after supported-client migration and retirement evidence.
• Retain enough artifacts for authorized rollback and replay; do not keep unneeded live endpoints forever.

Takeaway: A model upgrade is safe only when the interface and decision meaning understood by every supported consumer remain true throughout rollout, coexistence, fallback and recovery. Structural schemas are one part of the contract; semantic correctness and cross-version execution tests determine whether production is actually compatible.