How do you validate numerical parity and decision consistency when migrating production ML inference runtimes or hardware?

A production inference runtime migration is safe only when the candidate runtime produces acceptable predictions and business decisions for the same approved inputs, while also meeting latency, reliability, and operating-cost requirements. Moving PyTorch inference from CPU to GPU, exporting to ONNX Runtime, enabling TensorRT, changing framework versions, or switching FP32 to FP16/INT8 can alter numerical results without changing the API schema. Do not demand bitwise equality across platforms; instead define explicit numeric tolerances, decision-safety invariants, representative-data tests, and rollback conditions.

Curriculum boundary: Question 24 introduced broad environment parity, Question 26 lockfiles, Question 105 deterministic training-pipeline fixtures, and Question 120 immutable inference packaging. Question 133 discussed cost/performance optimization while Question 137 covered *API and prediction-contract evolution*. **Question 138 targets numerical and semantic equivalence of the *same intended model* when its inference backend, hardware, precision, serialization, kernels, or execution provider changes.** It does not repeat generic package pinning, schema evolution, or model-quality monitoring.

Beginner-friendly production incident

A delivery company serves a lateness-risk model with a business rule: send a package for manual review when predicted late-delivery probability is at least 0.70. The current runtime uses FP32 PyTorch on a CPU. An optimized runtime uses mixed-precision kernels on a GPU.

For the same normalized package input:

• Reference model produces 0.6998, so the business rule says *no manual review*.
• Candidate runtime produces 0.7003, so the business rule says *manual review*.
• Absolute probability difference = |0.7003 − 0.6998| = 0.0005.
• A numeric test with absolute_tolerance = 0.001 would consider this difference acceptable.
• Yet the business decision changed, so an invariant requiring zero unapproved decision flips would fail.

This is why numerical closeness and decision equivalence are separate deployment gates. For a low-impact warning, a small known flip rate might be acceptable after risk review; for consequential or irreversible decisions, the policy may require zero critical flips or independent escalation. Neither rule is universal.

Why same weights can produce different outputs

• Floating-point nonassociativity: (a+b)+c can differ from a+(b+c) because intermediate values are rounded. Different kernels or parallel reduction order can alter results.
• Precision changes: FP16, BF16, TF32, INT8, FP8 and other representations trade representable range, precision and compute/memory cost.
• Quantization: Mapping values into a finite integer range introduces rounding and clipping, and static calibration depends on representative data.
• Operator conversion: ONNX export, runtime graph optimizations, fused operations and execution-provider implementations may differ in supported semantics or numerical behavior.
• Preprocessing mismatch: Changing image resize interpolation, tokenizer version, missing-value imputation, feature units or categorical mappings changes *inputs* and is not just harmless numerical drift.
• Inference mode and randomness: Failing to switch a framework to evaluation mode, or leaving stochastic preprocessing/Monte Carlo layers enabled, can create substantial variation unrelated to conversion.
• Runtime and hardware versions: GPU architecture, drivers, cuDNN/kernel versions, threading, graph optimizations and available operators can change execution paths.
• Batch-dependent behavior: Dynamic batching, padding, sequence length, shape selection or incorrectly retained state may change results even if one-request golden tests pass.

PyTorch's numerical accuracy documentation states that bitwise-identical floating-point results are not guaranteed across platforms, and its deterministic-algorithm controls have limits. ONNX Runtime warns that quantization is not lossless and provides activation/weight comparison tools. NVIDIA recommends validation of intermediate layers, nonfinite outputs, calibration and precision choices for TensorRT.

Four acceptance dimensions

1. Structural validity: Input/output shapes, dtypes, label names, tensor layout, versioned schema and missing-feature behavior are correct. This is necessary but not sufficient.
2. Numerical parity: For the same canonical input, output differences remain within model-specific, predeclared absolute/relative tolerances. Track maximum, p95/p99, error patterns and any NaN/Inf.
3. Decision consistency: Score-to-action logic—including thresholds, top-k labels, abstention, risk bands, rules and fairness-critical subgroups—remains within an approved disagreement budget.
4. Operational safety: p95/p99 latency, throughput, memory, error rate, warmup time, model load, dependency compatibility, cost and rollback target meet production requirements.

If the source runtime itself is known to contain a defect, do not blindly use its outputs as the only gold standard. Compare with authoritative model-specification fixtures, separately validated expected behavior, matured ground truth and policy owners.

Numerical example 1: absolute and relative error

Suppose a reference model outputs r=0.8000 and a candidate returns c=0.7992.

• Absolute error = |c−r| = 0.0008.
• Relative error = 0.0008 / 0.8000 = 0.001 = 0.1%.
• If the policy allows atol=0.001 and rtol=0.001, the standard closeness form is |c−r| ≤ atol + rtol×|r|, or 0.0008 ≤ 0.0018, so this sample passes.
• But if r=0.00001, relative error is unstable as a sole metric. Use absolute error near zero and model-specific tolerance.
• Closeness tests alone cannot validate that both scores lead to the same *decision* at a threshold.

Use a carefully documented reference orientation: numerical libraries such as NumPy calculate closeness with the second argument treated as the reference in the relative term, so not all formulations are symmetric. For vector probabilities also check finite values, valid ranges and sum-to-one where the model contract requires it.

Numerical example 2: disagreement and model-quality gate

A representative labeled holdout contains 20,000 cases; the existing serving policy uses threshold 0.70.

• Candidate and reference yield the same business decision for 19,940 cases.
• They disagree for 60 cases; disagreement rate = 60/20,000 = 0.3%.
• Of those 60, 12 are actual positive late-delivery cases that were flagged by the reference but missed by the candidate.
• These 12 contribute 12/20,000 = 0.06% of *all evaluated cases*, but this is not the false-negative rate; that denominator must be actual positives.
• If the holdout contains 2,000 actual positive cases, these 12 represent an incremental 12/2,000 = 0.6 percentage-point loss in recall only if no other positive-case flips compensate. Confirm the full confusion matrices before claiming an actual recall change.

Decision: Evaluate disagreement type, false-negative cost, sensitivity groups, uncertainty and business thresholds. A 0.3% overall disagreement rate can conceal a severe problem concentrated in a small or high-risk cohort.

Numerical example 3: throughput gains versus correctness

Under identical representative traffic and hardware price assumptions:

• FP32 baseline: 100 valid on-time predictions/sec at $1.50/hour.
• INT8 candidate: 165 raw predictions/sec at $1.50/hour.
• If only 96% of candidate predictions satisfy validity/decision policy at that operating point, useful throughput is 165×0.96 = 158.4/sec.
• Baseline compute cost per 1 million useful predictions ≈ $1.50/(100×3600)×1,000,000 = $4.17.
• Candidate compute cost per 1 million policy-valid predictions ≈ $1.50/(158.4×3600)×1,000,000 ≈ $2.63.
• Despite theoretical cost savings, a 4% invalid/unauthorized-decision fraction is unacceptable for most strict inference policies; no cost improvement justifies bypassing the quality gate.

This is a hypothetical example. In practice, count actual eligible traffic, server-side errors, fallback rates, correctness and additional storage/dependency costs. The aim is cheap valid on-time decisions, not maximum raw tensor operations.

Numerical example 4: model score error near a boundary

Consider reference scores [0.10, 0.50, 0.6998, 0.9000] and candidate scores [0.1002, 0.4995, 0.7003, 0.8997] at threshold 0.70.

• Per-example absolute differences: [0.0002, 0.0005, 0.0005, 0.0003].
• Maximum absolute difference = 0.0005.
• Mean absolute difference = (0.0002+0.0005+0.0005+0.0003)/4 = 0.000375.
• One of four decisions changes, 25% of this deliberately tiny boundary-focused fixture.
• That is not evidence of a 25% production-wide disagreement rate; the test set was deliberately selected to expose threshold behavior.

Maintain two kinds of fixtures: representative traffic samples for population metrics and targeted edge cases for hard invariants.

Practical migration validation workflow

1. Freeze the true source of comparison

• Identify immutable model weight digest, preprocessing/tokenizer revision, feature schema, current serving runtime version and business policy.
• Capture reference inference outputs in evaluation mode with controlled seeds and documented hardware where relevant.
• Protect and version test fixtures; do not use arbitrary private production data without authorization.

2. Build a comparable candidate

• Export or compile the model and record graph operator set, optimization passes, precision, quantization/calibration inputs and execution provider.
• Pin candidate framework/runtime/compiler/CUDA-driver versions in a reproducible manifest.
• Confirm the model loads on the exact target platform, not only a developer workstation.

3. Verify the shape and data path first

• Compare preprocessing outputs before comparing model scores. Check units, normalization, sequence padding, channel order, dtype, tokenizer and missing-value behavior.
• Validate output names, class order and business rule version.
• If preprocessing differs, identify and fix that mismatch before tuning numerical tolerances.

4. Run layered comparisons

• Single sample, varied batch shapes and sequence lengths, normal/peak request sizes, null/empty features, rare categories and out-of-range input handling.
• Numeric closeness distribution with absolute and relative error and explicit NaN/Inf assertions.
• Business decisions near every threshold and material subclass; report disagreement and asymmetry between safe→unsafe and unsafe→safe outcomes.
• Compare matured-label quality on representative and high-impact cohorts; a similar score distribution does not prove equal accuracy.
• Use intermediate activations/weights where supported to locate the first stage of divergence.

5. Benchmark deployment behavior

• Measure p50/p95/p99 end-to-end latency, request deadlines, GPU/CPU memory, batch wait, throughput, startup/warmup and serving error rates.
• Run stress tests with realistic concurrency; a fast single-request benchmark may produce queueing or memory failures at production load.
• Check feature-store, logging and fallback paths. A new execution provider must not break existing recovery guarantees.

6. Apply release gates

• Block the release for invalid schema, disallowed model provenance, NaN/Inf, forbidden feature input changes, policy-critical decision flips, or failed SLOs.
• Tolerance values should be justified by model sensitivity and task risk; neither 1e-3 nor “99% match” is a universal safe threshold.
• If quality changes are intentionally approved, treat this as a model/policy change with new evaluation and approval, not as an automatically equivalent runtime swap.

7. Roll out and observe

• Use the existing staged rollout or shadow validation (Questions 122–124); compare actual release identity, numerical differences on matched eligible requests where privacy allows, and business outcomes.
• Protect against logging sensitive raw inputs; favor privacy-safe fixtures, aggregates and approved comparison samples.
• Keep a known-good, executable rollback bundle including model, preprocessing, runtime and feature contract.

Illustrative Python parity test

PYTHON
import numpy as np

THRESHOLD = 0.70
ATOL = 1e-3
RTOL = 1e-3

# Golden outputs from immutable, approved reference runtime.
reference = np.array([0.10, 0.50, 0.6998, 0.90])
candidate = np.array([0.1002, 0.4995, 0.7003, 0.8997])

assert reference.shape == candidate.shape
assert np.isfinite(candidate).all(), "Nonfinite candidate result"
assert np.all((candidate >= 0) & (candidate <= 1)), "Invalid probability range"

abs_error = np.abs(candidate - reference)
numerically_close = np.isclose(candidate, reference, atol=ATOL, rtol=RTOL)
decision_flip = (candidate >= THRESHOLD) != (reference >= THRESHOLD)

print("Maximum absolute error:", abs_error.max())
print("Closeness failures:", int((~numerically_close).sum()))
print("Decision flips:", int(decision_flip.sum()))

# Example critical policy: no unapproved decision flips on safety fixtures.
assert not decision_flip.any(), "Decision parity gate failed"

This code is a minimal educational fixture, not a sufficient production gate. Use a larger representative held-out set, separate threshold-focused cases, per-cohort metrics, versioned thresholds, checked data provenance and actual runtime inference execution. The example intentionally fails the decision gate even though all score differences can pass the numeric tolerance.

Example migration manifest (illustrative, not vendor syntax)

YAML
runtime_migration:
  model_digest: sha256:<approved-model-digest>
  reference:
    framework: pytorch
    device: cpu
    precision: fp32
    runtime_version: pinned
  candidate:
    framework: onnxruntime
    execution_provider: cuda
    precision: fp16
    runtime_version: pinned
  tests:
    require_identical_input_feature_contract: true
    numeric:
      absolute_tolerance: task_specific
      relative_tolerance: task_specific
      reject_nan_inf: true
    decision:
      critical_unauthorized_flips_allowed: 0
      evaluate_threshold_boundary_set: true
      evaluate_important_cohorts: true
    operational:
      check_p95_and_p99_latency: true
      check_cost_per_useful_prediction: true
      check_fallback_and_rollback: true
  rollout:
    immutable_artifacts: true
    stage_then_shadow_or_canary: true
    rollback_bundle_required: true

The manifest is a policy sketch, not an executable ONNX Runtime or TensorRT configuration. Store actual numeric tolerances and safety rules in an enforceable CI job with accountable approval.

Root-cause investigations and production incidents

Incident A: FP16 deployment flips high-risk decisions

• Observe: Candidate score errors are small overall but concentrated around the 0.70 warning boundary.
• Debug: Recompute matched features, inspect output probabilities and positive-class mapping; compare FP32 and FP16 layer activations.
• Fix: Keep sensitive operators in higher precision or change conversion/calibration after review; test decisions in boundary cases before a new canary.
• Prevent: Mandatory decision-flip and cohort-level parity fixtures, not just average absolute error.

Incident B: converted ONNX model works on CPU but fails on GPU provider

• Observe: Provider logs report operator fallback, shape errors, execution-provider differences or unexpected runtime slowdown.
• Debug: Check ONNX opset, operator support, driver compatibility, dynamic shapes and actual provider assignment. Verify the code is exercising the *intended* provider; silently falling back to CPU can make benchmarking misleading.
• Fix: Use supported graph/operators, configure providers explicitly, benchmark the intended hardware and pin the tested graph/runtime image.
• Prevent: Provider-assertion checks and representative shape tests in target-platform staging.

Incident C: INT8 quantization loses accuracy on rare traffic

• Observe: Average numeric error is small, but predictions on rare low-light images or extreme tabular ranges diverge.
• Debug: Compare activation distributions, calibration coverage, clipping and layerwise errors. ONNX Runtime provides matching utilities for quantized versus float graph activations.
• Fix: Use a representative calibration set or selectively leave sensitive layers unquantized, and rerun model-quality validation.
• Prevent: Cohort-aware quantization tests, not only overall throughput or MSE.

Incident D: multi-GPU batch inference is nondeterministic

• Observe: Repeated runs of identical inputs on the same release produce slightly different scores, especially in parallel reductions.
• Debug: Check inference/eval mode, random/stochastic operations, kernels, device/thread settings and input batching. PyTorch notes that deterministic settings can change performance and do not guarantee cross-platform identity.
• Fix: Use supported deterministic kernels if needed, eliminate unintended stochastic code, or define bounded numerical variance and decision invariants.
• Prevent: Same-runtime repeatability tests plus separate cross-runtime tolerance tests.

Incident E: GPU runtime uses different image preprocessing

• Observe: Prediction disagreement is far larger than FP32/FP16 error would explain.
• Debug: Compare preprocessed tensor hashes/shapes and image resizing, normalization, channel order, padding and tokenizer versions before comparing neural network layers.
• Fix: Ship the same pinned preprocessing or a formally approved equivalent transformation. Audit input-contract parity.
• Prevent: Golden preprocessed tensors and end-to-end fixtures for every supported input type.

Incident F: performance test passes while actual application regresses

• Observe: Candidate kernel throughput improves, but total inference p99 worsens and timeout count rises.
• Debug: Measure preprocessing, queueing, host-device copies, feature retrieval, dynamic batching and postprocessing.
• Fix: Tune the real serving path, not only microbenchmarks; check model warmup and request-size tails.
• Prevent: End-to-end load SLO gates together with numerical parity.

Incident G: rollback of runtime engine fails

• Observe: Previous model weights remain but old container/driver/feature mapping was removed during migration.
• Debug: Compare immutable release components, image availability, startup logs, feature compatibility and model execution provider.
• Fix: Restore the entire known-good serving bundle. Do not claim model-only rollback is safe.
• Prevent: Test an actual reverse rollout before authorizing deletion of earlier artifacts.

Mode-specific risks

• Online inference: Threshold flips and p99 latency failures can affect business actions immediately. Keep strict on-time decision gates and per-release attribution.
• Batch inference: A runtime change can alter scores on millions of records in one run. Validate representative full partitions, deterministic output manifests and downstream decision thresholds before publication.
• Asynchronous prediction: Pin model/runtime identity per logical job or explicitly approve migration; retries may run after new code is deployed. Preserve idempotency.
• Streaming inference: Replays and checkpoint restores may execute with different hardware/runtime; compare event-time semantics, state transitions, output scores and duplicate side effects. Record the actual engine for each committed event or batch.
• GPU mixed-precision inference: Measure memory, dynamic batch effects, calibration, provider fallback and runtime-specific nonfinite values; accelerator utilization alone is not proof of correct decisions.

Observability after migration

Track:

• Per-release numeric checks: max and p95/p99 absolute/relative differences on governed paired samples; nonfinite scores, range violations and provider assignment.
• Decision safety: Threshold flips, direction of flips, important slices, abstentions, missing-feature paths, approved fallback use and downstream actions.
• Model quality: Matured-label precision/recall/calibration or task-specific metrics with confidence/coverage caveats; an unlabeled score shift is not definitive accuracy degradation.
• Serving health: p95/p99 latency, throughput, queue age, GPU/CPU memory, driver failures, allocation or kernel errors, startup and cold-load time.
• Provenance: Exact weights/graph digest, preprocessing and runtime image, execution provider, accelerator/driver version and business policy revision.
• Economics: Cost per valid on-time prediction including warm capacity, hardware, dependency and observability cost.

Use retained, privacy-approved replay fixtures and clearly separate *paired synthetic tests* from production behavior. A live canary may see a different customer mix; control traffic assignment before inferring causality from aggregate score distributions.

Trade-offs and limitations

• Bitwise equality: Strong in a fixed deterministic runtime, but commonly unrealistic across architectures and versions; forcing it may add unnecessary latency and still miss semantic input defects.
• Numeric tolerances: Necessary for heterogeneous hardware, but an overly loose tolerance can hide threshold-crossing user harm.
• Decision parity: Strong protection at current rules, but zero flips across a small fixture set does not prove broad model-quality equivalence.
• INT8 and mixed precision: Often reduce memory/compute but can increase quantization or rounding error; cost gains are workload/hardware dependent.
• Per-layer debugging: Reveals divergence source but can be expensive for large models and graph-fused operators.
• Representative evaluation: Measures typical population behavior; rare high-consequence cohorts need separate targeted coverage.
• Canary observation: Catches runtime and business integration issues, but delayed outcomes can prevent immediate quality conclusions.
• Deterministic algorithms: Improve reproducibility and debugging but may slow inference or be unsupported for some operations.
• Cross-region migration: Different GPU types or driver stacks can violate assumption of identical outputs; require per-target-platform evidence.

Common production mistakes

• Comparing only model weights or file hashes and assuming numerically equivalent execution.
• Calling every nonzero floating-point difference a bug, or ignoring tiny differences near business thresholds.
• Choosing 1e-3 as a universal safe tolerance without evaluating decision risk.
• Measuring only average score error while ignoring rare cohorts and false-negative direction.
• Running in training mode during inference comparison.
• Comparing outputs generated from different normalization or tokenizer revisions.
• Testing a single batch size but deploying dynamically batched traffic.
• Assuming an ONNX export implies every operator runs on the intended GPU provider.
• Equating a passing HTTP schema test with semantic prediction parity.
• Approving an INT8/FP16 swap based only on latency or memory improvements.
• Deleting old runtime dependencies before verifying full-stack rollback.
• Averaging error percentages near scores of zero, creating misleading relative metrics.
• Counting only on-time requests while ignoring accuracy, invalid scores or unauthorized decisions.

Production release acceptance checklist

• Reference and candidate have immutable model/graph/runtime/preprocessor IDs.
• Target hardware, driver, execution provider and precision are recorded and tested.
• Same canonical inputs, preprocessing, feature contract and business policy are used for paired comparisons.
• Test set includes representative traffic, high-impact cohorts, threshold boundaries and pathological inputs.
• Numeric closeness, NaN/Inf, score range, label order, semantic decisions and matured-label model quality each have explicit acceptance criteria.
• End-to-end load/SLO and cost tests pass for the actual deployment topology.
• Canary or shadow measurements preserve per-version provenance and accurate attribution.
• Fully executable previous release remains available and rollback has been rehearsed.
• Deviations are approved as intentional model/policy changes, not silently ignored as numerical noise.

Takeaway: Cross-runtime parity is not “does the model return a similar number?” It is does the approved model still make sufficiently consistent, valid and on-time decisions on the target hardware with a reproducible, reversible release? Numeric tolerances, risk-aware decision checks and real deployment tests must all pass before promotion.