Large Language Models
Large language models are deep learning models, usually transformer-based, trained to predict tokens from large corpora. After pretraining, they are often instruction-tuned and aligned so they can follow prompts and assist users.
How do you build a continuous inference configuration evaluation pipeline that detects quality, latency, and cost regressions before production deployment?
A continuous LLM inference configuration evaluation pipeline turns every model, runtime, GPU, quantization, tokenizer, batching, caching or routing change into a reproducible experiment with an automated release decision . It joins functional/quality tests, realistic load benchmarks, infrastructure cost estimates and failure envelope checks; preserves immutable evidence; and blocks unsafe changes before any production promotion . This is different from Question 266 (debugging a regression after a model/runtime/hardware upgrade), Question 258 (how to design a single realistic serving benchmark), Question 264 (canary rollout and rollback) and Question 280 (how to weigh performance, quality, reliability and cost across candidates). This lesson specifically designs the continuous engineering system: trigger graph, fixture provenance, CI runners, baseline storage, per change test selection, reproducible comparison, release gates, artifact signing and the handling of uncertain results. 1. The challenge: a green unit test suite is not sufficient An inference container may start successfully and pass API schema tests while: Its kernel/library version changes GPU throughput or produces CUDA failures on a particular GPU. Quantization changes a critical model's generated answer quality or structured output adherence. New tokenizer or prompt templates alter token counts and invalidate cached prefix savings. Increased \max num batched tokens\ raises output throughput but harms p95 time to first token (TTFT). Aggressive KV allocations improve ordinary load yet cause out of memory failures on long context requests. A cheaper GPU configuration misses peak SLOs or loses reliability when one replica fails. Prefix caching or retry policy makes a benchmark appear cheaper but increases real failed work. An effective gate must catch behavioral and systems regressions , including interactions, not just crashes or raw token/second changes. 2. Pipeline architecture and triggers Use the following stages: 1. Change detection: trigger on Git pull requests, new container/driver versions, model weight or adapter hashes, tokenizer/chat template changes, GPU SKU migrations, quantization settings, autoscaler and serving configuration updates. 2. Reproducible build: assemble an image by immutable digest and a configuration manifest with the exact model/runtime/driver/hardware and decoding settings. 3. Cheap deterministic tests: validate config schema, model/adapter compatibility, health endpoints, tokenizer behavior, API shape, authentication, safety constraints and maximum resource requests. 4. Functional/quality evaluation: run a stratified versioned golden dataset, critical regression examples, safety cases, tool/JSON checks and stochastic repeat trials where needed. 5. Representative serving tests: execute realistic offered load arrival patterns and prompt/output mixes; capture client observed TTFT, inter token latency (ITL), full stream completion, p95/p99, SLO goodput and all request failures. 6. Cost and capacity calculation: estimate total GPU hours, required replicas for safe load, actual startup overhead and spend per successful SLO compliant original request using a versioned rate card. 7. Failure envelope validation: test an isolated worker restart, a cold replica, KV pressure and N−1 capacity in dedicated nonproduction infrastructure. 8. Release decision: compare against explicit baselines and hard policy gates. Save \PASS\, \FAIL\, \INCONCLUSIVE\, or \REQUIRES APPROVAL\ with reasons. 9. Promotion record: attach benchmark artifacts, approval, code/model digest, evaluation data versions and observed risk; release manager verifies evidence before staged rollout. 10. Continuous drift follow up: rerun scheduled benchmarks against a control baseline and evaluate sampled, privacy compliant production traces so changes in the environment are noticed. Frequency: fast tests on every code change; selected representative GPU tests for relevant PRs; more expensive peak load/failure tests on release candidates; scheduled, identical hardware control runs to detect environmental drift. 3. Identify and pin every independent variable An experiment fingerprint should include: Model weight hash, model repository revision, adapter IDs and their digests. Tokenizer hash, chat template, system prompt policy, sampling settings, \max tokens\, context length and tool schema. GPU SKU and count, MIG/partitioning profile if relevant, topology, driver/CUDA/library versions. Runtime and container digest; tensor parallel degree, attention backend, quantization and KV dtype. Scheduling config (\max num batched tokens\, active sequences, chunked prefill), routing, caching and model warmup status. Workload replay version: arrival process, seed, prompt/output length distribution, dataset permissions and tenancy distribution. Evaluator code and judge version, human labeled reference data, quality rubric, metrics window and price sheet. Do not compare a candidate against a baseline with different GPU hardware, warm cache state or workload mix and call the difference a code regression. If hardware changes are the point of the experiment, compare end to end feasibility and cost across the separately benchmarked hardware, not a falsely normalized percentage difference. 4. Beginner friendly numerical example: stop a deceptive win Assume the baseline serving setup is tested at 10,000 original requests under an approved traffic mix: p95 TTFT = 700 ms , p95 ITL = 38 ms , quality pass rate = 96% , joint SLO completion = 9,600 , and modeled GPU cost = $40 for the benchmark run. Baseline cost per joint SLO compliant request = \$40 / 9,600 ≈ $0.00417\. A candidate changes batching and KV settings and reports: +18% output tokens/s , p95 TTFT = 1,100 ms , p95 ITL = 37 ms , quality = 96% , joint SLO completion = 9,200 , and GPU cost = $36 . Candidate cost per joint SLO compliant request = \$36 / 9,200 ≈ $0.00391\, roughly a 6.1% lower cost per good request . Suppose the hard interactive TTFT gate is p95 ≤ 900 ms and joint success minimum is 95% of original offered requests: Baseline passes: 700 ms and 96%. Candidate fails : 1,100 ms and 92% (\9,200 / 10,000\). Correct CI decision: FAIL , despite higher raw throughput and lower apparent unit cost. Statistics caveat: 96% quality here is from a separate curated evaluation set; it must not be treated as the exact same quantity as 96% joint SLO completion. Define each dataset and denominator explicitly. All numbers are illustrative, not vendor benchmarks. 5. Quality gate design: deterministic plus judgment based Deterministic checks: JSON/schema validity, permitted tool names/arguments, prohibited output patterns, refusal/safety constraints and known answer exact matches where appropriate. Tool calls in shadow evaluation must use mocks or safe test harnesses; never execute real money moving or destructive actions. Evaluate critical product tasks separately: RAG groundedness, citations, code pass@tests, classification or extraction correctness and long context evidence use. Probabilistic checks: Repeated runs for nondeterministic outputs; paired prompts between baseline/candidate. Automated model judges scored against a human verified sample, not presumed infallible. Stratified datasets by difficulty, prompt length, language, tenant class, context type and failure mode. Apply predeclared noninferiority margins and confidence intervals to meaningful differences. Do not declare a model change safe just because the mean of an LLM judge score is unchanged. A critical safety category can regress despite a stable average. If a slice is under sampled, mark it \INCONCLUSIVE\ or require targeted additional tests. MLflow documents repeatable GenAI evaluation, regression tests integrated with ordinary \pytest\, versioned datasets and comparison runs. 6. Latency and throughput benchmarks in CI Benchmarks should: Use open loop offered arrivals for realistic load whenever feasible. Closed loop clients can hide overload by slowing arrivals as responses become slow. Include interactive bursts, a long context tail, output length variation, cancellation, mixed batch/chat priorities and enough completed and failed requests. Measure client side TTFT, ITL/TPOT, end to end completion, request errors and SLO goodput; never aggregate p95s across shards by simple averaging. Track GPU memory, KV cache occupancy/preemptions, engine queueing, worker restarts and CPU/network bottlenecks. Include both cold and warmed runs; preserve cache state in experiment metadata. Do not reuse the baseline's cache for one candidate and not the other. Avoid a load generator bottleneck; monitor generator CPU/network and achieved offered rate. vLLM provides serving benchmarks and a continuous benchmark dashboard with scheduled serving/throughput/latency tests. NVIDIA AIPerf supports multiple arrival patterns, percentile TTFT/ITL and output throughput, and trace replay. Their tooling produces evidence; the application still needs its own quality and cost gates. 7. Two tier CI: fast path versus expensive GPU gate Fast path (every PR): 1. Lint, config validation, dependency/license scanning and basic model load smoke tests. 2. A small, high value deterministic and critical quality dataset. 3. Lightweight functional and format stability tests. 4. Configuration diff classification: if only docs changed, avoid an unnecessary expensive GPU run. Expensive release candidate GPU path: 1. Allocate an approved, comparable GPU runner pool via a quota aware job scheduler. 2. Warm the baseline and candidate under the same recorded protocol. 3. Run realistic open loop TTFT/ITL and joint SLO goodput load points. 4. Run exhaustive quality/safety slices and important failure envelope cases. 5. Save machine readable run artifacts; free runners after confirmed completion. 6. Block promotion until required tests pass or a documented exception decision is issued. Risk based test selection: a tokenizer, runtime, quantization or model weight change gets a deeper suite than a logging only edit. However, shared dependencies or kernel/runtime upgrades may affect many models; periodic broad matrix tests catch omissions in dependency based selection. 8. Illustrative GitHub Actions workflow The snippet below describes orchestration only . It requires a properly installed self hosted GPU runner , a real workload test program and centrally managed model/artifact access. \\\yaml name: inference evaluation on: pull request: paths: "serving/ " "models/ " "eval/ " workflow dispatch: jobs: fast checks: runs on: ubuntu latest steps: uses: actions/checkout@v4 run: python m pip install r eval/requirements.txt run: pytest q eval/tests/smoke gpu release gates: needs: fast checks runs on: [self hosted, linux, gpu eval] timeout minutes: 180 concurrency: group: inference gpu evaluation cancel in progress: false steps: uses: actions/checkout@v4 run: python m pip install r eval/requirements.txt name: Evaluate and compare candidate run: python eval/compare configurations.py policy eval/release policy.yaml uses: actions/upload artifact@v4 if: always() with: name: inference evaluation results path: eval/results/ \\\ Deployment caveats: GitHub Actions job level \concurrency\ must be a sibling of uns on\, \steps\ and \timeout minutes\ (as shown). The example's commands/scripts are project specific placeholders; they do not exist by default. Pin marketplace action versions/SHAs and review permissions/supply chain exposure before production adoption. A unique request level run ID must be in generated evaluation artifacts. Dedicated GPU runners should be ephemeral or properly isolated from untrusted pull request code. 9. The artifact contract Each run emits immutable, indexed artifacts : \manifest.json\: precise configuration fingerprint, model and image digests, dataset/evaluator revisions, hardware and pricing. \quality.json\: per item results, per slice metrics, confidence intervals, grader version and safety rule outcomes. \performance.json\: offered arrivals, request trace identifiers, p50/p95/p99 TTFT, ITL, E2E latency, errors, joint goodput and GPU telemetry. \cost.json\: billed/estimated hardware time, GPU price basis, retries, warmup/standby assumptions, cost per validated SLO request and uncertainty. \decision.json\: \PASS/FAIL/INCONCLUSIVE/EXCEPTION REQUIRED\, policy revision, reasons, owner approvals, protected metrics and links to baseline. \logs and traces\: appropriately redacted diagnostic samples, with retention and access rules. Keep enough raw request level data to recompute percentiles rather than only storing aggregate p95. Protect private prompts/customer traces by using consented and sanitized evaluation datasets; don't publish them in public CI logs. 10. Gate implementation and failures A conceptual comparator: \\\python def release gate(baseline, candidate, policy): if not comparable(baseline, candidate): return "INCONCLUSIVE: mismatched hardware/workload" if not candidate.telemetry complete or candidate.n requests < policy.minimum samples: return "INCONCLUSIVE: missing evidence" if not candidate.security and safety pass: return "FAIL: mandatory policy" if not candidate.critical quality slices pass: return "FAIL: critical quality" if candidate.p95 ttft ms policy.max p95 ttft ms: return "FAIL: TTFT" if candidate.p95 itl ms policy.max p95 itl ms: return "FAIL: inter token latency" if candidate.joint slo success < policy.min joint success: return "FAIL: joint goodput" if not candidate.n minus one pass: return "FAIL: resilience" if candidate.quality difference uncertain: return "INCONCLUSIVE: repeat evaluation" return "PASS: eligible for guarded promotion" \\\ Never automatically approve a candidate on missing data or a benchmark with zero valid completed requests . A static test can fail deterministically; near boundary statistical results should follow a documented confidence/noninferiority procedure. Cost improvement is evaluated only among candidates satisfying hard policy thresholds . 11. Production incident: performance passes, answer quality silently drops Hypothetical incident: An engine upgrade improves throughput by 15% in a nightly benchmark. The release pipeline checks only latency and output tokens/s; the deployment ships. A rare long context retrieval use case loses citation fidelity after a tokenizer/template interaction, even though the model still responds normally. Root cause: 1. Model evaluation and GPU performance tests were maintained in separate pipelines, with no joint deployment gate . 2. The golden dataset lacked long context evidence tasks and citation quality slices. 3. The benchmark manifest omitted chat template/adapter hashes, so rerunning the old “baseline” did not recreate its actual behavior. 4. The performance job measured warm repeated prompts, not production like cache variation. Recovery: rollback to an immutable known good manifest; add long context groundedness and citation checks; store full configuration fingerprint; replay quality and load tests against the identical target setup. Prevention: a single signed release decision artifact must point to both quality and systems evaluation plus appropriate failure tests; a missing or inconclusive required check must block promotion. 12. Distinguish continuous regression from normal noise Use repeated baseline and candidate runs interleaved or paired where practical, not single isolated tests on differently loaded hardware. Exclude or control external noisy neighbor GPU jobs and environmental shifts. Track sample size, confidence interval and measurement error; avoid overreacting to a tiny p95 difference from few requests. Run comparison by workload slice . A fleet wide average can hide a high priority tenant regression. Maintain long lived reference baselines and a separate rolling “last known good” candidate; both have value. Treat an improvement as tentative until confirmed by matched test conditions and sufficient quality coverage. Avoid automatically overwriting the baseline with the current release, or a sequence of small regressions will be normalized into acceptance (“baseline drift”). 13. Production gate and post release feedback Passing offline evaluation means eligible for a controlled release , not immediately approved for 100% traffic. Apply the previously established shadow/canary rules, enforce streaming success and per tenant SLO alerts, and monitor delayed quality signals. On any failure, rollback to the last approved immutable configuration. Turn each confirmed production failure into: A new sanitized regression test case and associated affected workload slice. An updated policy threshold or stronger run artifact requirement where necessary. A replay/load test condition reproducing the failure envelope. A code or configuration fix and a rerun whose evidence is linked to the incident. This feedback loop improves the pipeline without relying on training data leakage, public benchmarks alone or unreproducible “looks good” outputs. 14. Engineering trade offs Coverage versus GPU CI cost: broad GPU matrices are expensive; use changed component risk selection for PRs and scheduled exhaustive sweeps, not arbitrary test skipping. Latency to merge versus safety: cheap fast checks can run per commit; deep release gates may be slower but must not be bypassed silently. Determinism versus realism: fixed seeds and curated prompts help debug, while diverse randomized workloads expose real variations. Use both and preserve seeds. LLM judges versus human review: automatic judges scale but can be biased. Calibrate with humans, especially for critical failure slices. Temporary GPU shortages: if a required runner is unavailable, emit \INCONCLUSIVE\; do not reinterpret “tests did not run” as “tests passed.” Expensive model or runtime startup: reuse immutable artifacts where safe while keeping candidate and baseline conditions comparable. Confidentiality: protect datasets and request traces; prohibit untrusted code from seeing production secrets. 15. Interview ready answer “I would make every inference relevant change trigger a versioned evaluation pipeline. Fast CI checks configuration and functional safety; release candidate GPU jobs run quality and realistic serving benchmarks with pinned model/engine/GPU/workload versions. A decision engine compares candidate to a known good baseline on critical quality slices, p95 TTFT/ITL, original request joint goodput, N−1 recovery and cost per successful outcome. It saves immutable machine readable artifacts and blocks promotion on failures or missing evidence. A guarded canary and ongoing regression jobs feed new production failures back into the golden suite.” Research references: vLLM — Performance Dashboard and Continuous Benchmarking vLLM — Serving Benchmark CLI NVIDIA — AIPerf for Realistic LLM Load MLflow — Regression Testing and CI/CD MLflow — Continuous LLM Evaluation Guide (2026) GitHub Actions — Workflow Syntax
How do you detect benchmark drift and prevent false performance regression alerts in continuous LLM inference evaluation?
Detecting Benchmark Drift and False Regression Alerts in Continuous LLM Inference Evaluation Benchmark drift occurs when the test environment, traffic mix, instrumentation, inputs or reference baseline changes enough that repeated results are no longer strictly comparable. False performance regressions occur when an evaluation pipeline attributes variation to the candidate model/engine/configuration even though the observed change is caused by noise, environment differences, measurement defects or an inappropriate comparison. A good continuous inference evaluation system answers two separate questions : 1. Comparability: did the candidate and baseline run under equivalent conditions, or is this an invalid comparison? 2. Regression evidence: after controlling known differences, is there convincing and practically important evidence that the candidate violates a previously specified performance or quality requirement? Prerequisite distinction: Question 258 covers constructing representative performance tests, Question 266 covers diagnosing true production regressions after upgrades, and Question 281 covers wiring an automated evaluation pipeline. This lesson specifically covers repeated run variability, reproducibility controls, benchmark drift classification, statistically defensible alert rules, and suppression of misleading release blocking alarms. 1. Why LLM serving benchmarks drift A serving benchmark is sensitive to more than the code revision: Workload drift: changed prompt length distribution, output token caps, system prompts, tool schemas, streaming mode, tenant concurrency or arrival process. Warmup drift: initial weight loads, CUDA graph compilation, first request allocation, runtime autotuning and cache state. Mixing a cold first run with warmed steady state distorts TTFT. Hardware drift: a different GPU SKU, memory configuration, MIG partition, NVLink topology, CPU/socket, driver, CUDA runtime or firmware. Resource interference: unrelated GPU jobs, CPU contention, storage/model download traffic, thermal/power throttling and cross node network congestion. Benchmark client drift: insufficient load generator workers, pacing jitter, CPU saturation, incorrect request scheduling, connection reuse or timeouts. Software dependency drift: tokenizer, inference backend, attention kernels, drivers or decoding defaults changing without a corresponding tracked candidate change. Metric definition drift: measuring per request time per output token (TPOT) versus inter token gap (ITL), mixing percentiles, or dropping errored/timed out requests from the denominator. Dataset/cache drift: regenerated synthetic prompts, public dataset revisions, PII sanitization changes, cached prefixes shared across test runs, prompt duplication or a different random seed. Statistical noise: limited request counts, heavy tailed context lengths, nondeterministic decoding and noisy p99 estimates. A raw number such as “p95 TTFT grew by 8%” is not enough to identify its cause. Evidence from research A Microsoft Research paper published in October 2026 , Reproducible LLM Inference Benchmarking: A Sequential Isolation Protocol for Regression Testing , evaluated three open source models on one A100 GPU using vLLM 0.9.1, six context sizes, eight concurrency levels and five repetitions per configuration. Across its study, a sequential isolation protocol reduced average coefficient of variation from 15.2% to 2.2% . That is study specific evidence , not a universal expected noise floor for every cluster, model, tail metric or workload. Source: https://www.microsoft.com/en us/research/publication/reproducible llm inference benchmarking a sequential isolation protocol for regression testing/ 2. Classify the benchmark outcome before alerting Use explicit classifications: 1. TRUE REGRESSION: reproducible and material candidate degradation at comparable workload and valid measurement, or a confirmed violation of a hard SLO. 2. FALSE ALARM OR NOISE: initial result cannot be repeated under controlled conditions; no credible evidence of material regression. 3. ENVIRONMENT DRIFT: test hardware, system state, interference, runtime or cache changed outside the intended variable. 4. WORKLOAD DRIFT: request/length/arrival distribution differs; the two runs answer different performance questions. 5. MEASUREMENT FAILURE: generator underdelivered offered QPS, incomplete trace capture, corrupted timestamps, missing telemetry or broken success accounting. 6. INCONCLUSIVE: confidence interval/sample size is too weak; required repetitions or isolation tests remain incomplete. 7. INTENDED BEHAVIOR CHANGE: candidate intentionally changes output lengths or decoding; compare expected policy and quality adjusted service outcomes rather than interpreting every metric movement as a regression. Important: a failed comparability check is not permission to promote . Mark it INCONCLUSIVE or INVALID, rerun appropriately, and block required deployment gates until valid evidence exists. 3. Beginner friendly numerical example: noise versus real slowdown Suppose your CI measures median time to first token (TTFT) in five independent matched benchmark repetitions . In each repetition, replay the same workload manifest, offered arrival rate, GPU topology and warmup protocol. Baseline median TTFT, in milliseconds: 190, 194, 188, 192, 196. Mean of these repetition level medians = 192 ms . Sample standard deviation ≈ 3.16 ms . Coefficient of variation ≈ 1.65% . Candidate repetition level median TTFT: 193, 198, 191, 196, 199. Mean ≈ 195.4 ms . Nominal change ≈ \(195.4 − 192)/192 × 100 ≈ 1.77%\. A bad gate says “candidate baseline by 1% → FAIL” based on one noisy repetition. A better gate first checks comparable environment and studies the paired differences on matched repetitions: +3, +4, +3, +4, +3 ms. The candidate appears consistently slower by roughly 3–4 ms on this controlled example, but whether that should block release depends on a prespecified practical regression margin , the test design and inferential uncertainty. For instance, if the product's meaningful regression threshold is +10 ms and the hard p95 TTFT SLO is comfortably met, the observed ~3.4 ms shift may be statistically detectable while still being operationally acceptable . Do not confuse “no materially unacceptable degradation” with “the implementations are numerically equal.” Contrast: materially bad candidate: paired differences of +28, +32, +29, +31 and +30 ms. This is larger than a 10 ms materiality margin and consistently observed. With valid instrumentation and repeated runs, the pipeline can present strong evidence of a meaningful slowdown. Statistical caution: the example uses means of repetition level medians , not pooled per request median, p95 or p99; no claim about tail latency follows from these numbers. Five repetitions can be insufficient for production p99 evaluation. Predeclare the estimand, sample size and decision rule. 4. What to freeze in the benchmark manifest For every run, write an immutable fingerprint including: Model and adapter digest, tokenizer/chat template hash, generation parameters, maximum output/context tokens, safety/filter configuration. Engine/container digest, serving scheduler configuration, attention backend, quantization, tensor parallelism, prefix cache, KV dtype, compiled kernels and CUDA/driver version. GPU SKU/count/topology, CPU host, memory, cgroup quotas, Kubernetes node/zone, MIG partition and other running workloads. Test runner version, load generator hardware/resources, endpoint URL/protocol, connection reuse, streaming setting and pacing strategy. Synthetic or replay dataset snapshot/hash, request IDs, seed, offered arrival schedule, prompt/output length strata and any data scrubbing transform. Warmup policy, measured interval boundaries, model loaded readiness, cache warm/cold declaration, and benchmark start/end timestamps. Metrics schema/version, clock source, valid/error/timeout request accounting, client generated/achieved request rate and raw response trace. Price basis and cost attribution version if a cost regression is tested. A content hash of the entire manifest makes it easy to identify accidental baseline mismatch. Do not quietly compare two dissimilar manifests as though the candidate changed only one serving knob. An intentional hardware/SKU comparison can still be useful, but its question is overall trade off at separately measured operating points , not causal attribution of a software regression. 5. Cold start and warm steady state are distinct experiments Benchmark both deliberately: Cold start experiment: includes image/model download, allocation, compilation and first request. Report time to ready and first request latency. Warm steady state experiment: runs a documented warmup phase, then measures normal serving after state stabilizes. Warm prefix cache experiment: replays a realistic repeat prefix distribution, if relevant to production. Cold prefix cache experiment: uses enough distinct prefixes or clear cache isolation to prevent accidental cache reuse. If the baseline had hot cache but the candidate begins cold, a large TTFT difference is a cache state confounder . Conversely, clearing cache can hide a real prefix cache optimization that benefits production. NVIDIA AIPerf explicitly supports pre test warmup settings and controllable arrival patterns. Its load options can request Poisson, gamma and constant arrival distributions, as well as fixed schedule trace replay. References: https://docs.nvidia.com/aiperf/tutorials/load patterns scheduling/warmup phase configuration https://docs.nvidia.com/aiperf/benchmark modes/load generator options reference 6. Avoid load generator artifacts A benchmark may “regress” because the benchmark client is saturated rather than the GPU: 1. Record requested versus actual scheduled arrivals per time window. 2. Record generated requests, completed requests, errors, cancellations and timeouts separately . 3. Measure client CPU, memory, socket limits, network throughput and clock behavior. 4. Inspect schedule lag distribution; do not let closed loop client backpressure silently reduce offered load. 5. Use independently provisioned generators when the target is faster than a single test process. 6. Do not treat low successful throughput during high errors as acceptable completion. 7. Verify request text, token length and output token stop criteria are equivalent. NVIDIA notes that high QPS benchmark pacing can be limited by event loop timer granularity; its load generator documentation discusses higher resolution pacing. AIPerf's multiprocess architecture is intended to reduce client bottlenecks. Production relevant caution: correct observed QPS does not by itself establish correct open loop arrivals. Bursty Poisson and near constant arrivals can yield very different queueing tails at the same average offered rate. 7. Use paired experiments to isolate candidate effect A robust comparison procedure: 1. Choose an approved reference hardware/runtime envelope and verify GPU/CPU/network health. 2. Run a baseline A under that envelope. 3. Recreate the same workload and run candidate B ; where feasible use an ABBA or randomized A/B order across repetitions to reduce drift from time order effects. 4. Reserve comparable warmup and cooldown windows; avoid hidden KV/prefix cache carryover between variants. 5. Record independent repetition level results, not just millions of correlated tokens from one run. 6. Compare p95/p99 latency and success from per request data; use matched prompt slices and repeated seeds where appropriate. 7. If a reference control A drifts when the candidate did not change, classify the environment as suspect. 8. Repeat in isolation if GPU thermal/power, load generator health, storage, node scheduling or noise differs. Repeated measurements inside one saturated benchmark are often correlated. A simplistic confidence interval treating each token as an independent observation understates uncertainty. Resample by independent run, session or appropriate arrival block, or use a dependence aware method. 8. Statistical gates without alert fatigue Build three separate gates: A. Hard production requirements: Reject confirmed quality/safety/region violations immediately. Reject a correctly measured TTFT/TPOT or availability SLO violation according to the predeclared policy. For uncertain near boundary results, repeat and report INCONCLUSIVE. Count every offered original request in success/error metrics, including failures. B. Material regression thresholds: Example: alert when p95 TTFT rises more than 10% and at least 25 ms , with adequate sample count, comparable traffic and reproducible evidence. Those thresholds are illustrative . Choose limits based on product SLO budget, benchmark noise and engineering impact. For cost, use $ per completed, quality acceptable SLO original request , controlling for model, GPU price, workload and retries. C. Statistical evidence: Use repeated matched runs and confidence intervals for the candidate baseline difference or ratio. Consider a minimum effect threshold , not only a p value. For a noninferiority claim, specify the tolerated degradation and test whether the appropriate one sided confidence bound stays within that margin. Predeclare treatment of p95/p99, multiple workload slices, and multiple comparison corrections or hierarchical screening as appropriate. Do not claim “no regression” merely because a low powered significance test failed to reject a null hypothesis. Practical distinction: “not statistically significant” does not prove equivalence. If the uncertainty band spans both acceptable and harmful outcomes, report INCONCLUSIVE and collect more data. 9. p95 and p99: sample size and aggregation traps Suppose a run has 100 completed requests . The “p99” is driven by roughly the slowest one or two observations, so a single noisy request can move the estimate sharply. At 10,000 requests it is more stable, but results can still be dominated by changed workload mix, temporally clustered failures or shared upstream variance. Common pitfalls: Never average p95/p99 of separate worker pools to form a fleet p95/p99. Merge raw observations or merge compatible histograms/sketches at the correct weights. Do not compute p95 from completed successes only and ignore failures/timeouts. Report SLO compliant goodput using all offered original requests. Compute per request TPOT according to your defined metric. Do not silently substitute token to token ITL for TPOT or vice versa. Split by short, medium and long context , different models and critical tenant classes. An aggregate p95 can improve while long context p95 gets much worse. A changed output length distribution changes decode work and apparent latency; compare request length stratification and, when meaningful, normalized metrics without concealing customer visible latency . Benchmarking metric definitions can differ across tools. Keep a versioned metric dictionary and test it whenever the test runner changes. 10. Reference baselines: fixed control versus rolling production Maintain two complementary references: Pinned known good baseline: an approved immutable deployment and workload fixture that is rerun on reference GPUs. Helps detect machine/test drift or slow, cumulative regressions. Recent deployed baseline: the currently approved live version on compatible hardware. Useful for an incremental release decision. Failure mode: always replacing the baseline with last week's “passing” result can permit ten successive 1–2% slowdowns to accumulate unnoticed. Compare to both an immutable historical anchor and the most recent acceptable release. Environment control case: if the pinned baseline's p95 TTFT has become 20% slower on unchanged code, the test lane may have changed. Investigate thermal throttling, GPU driver, runner quota, queueing, cache or pacing before blaming the candidate. Intentional baseline migration: moving from A100 to H100, or a new engine scheduler, is not necessarily a regression. Establish a new reference through a controlled migration and preserve the legacy baseline and reasons for discontinuity. 11. Drift detection and automated alert policy Track four independent fingerprints/time series: Environment drift GPU type/topology, temperature/clock throttling, host noise, driver/CUDA, runner/cgroup shape, CPU and storage/network performance. Rule: a different fingerprint is comparability failure unless the changed factor is explicitly under test. Workload drift Input/output length quantiles, request mix, offered arrival/burstiness, seed/fixture IDs, prefix cache hit pattern and cancellation ratio. Rule: an exceeded distribution distance threshold or major version mismatch should request a controlled replay. Measurement drift Offered versus achieved QPS, generator schedule lag, clock/timestamp consistency, error/timeout omissions, counter resets and export format changes. Rule: benchmark is INVALID if measurement coverage or timing is untrustworthy. Candidate performance drift Reproducible change in raw TTFT/ITL, joint SLO goodput, GPU hours per successful output, KV preemptions or error rates, after environment and workload comparability pass. A practical controller combines change point alerts, run control deviations and independent repetitions. Use alert severities: Informational: metric shifts within normal noise. Needs rerun: meaningful but uncertain difference. Block release: confirmed hard violation or required evidence unavailable. Investigate benchmark lane: reference control also regressed or manifest invalid. Critical operational: production SLO failing; follow production incident procedures rather than just benchmarking alerts. 12. Concrete automated comparator pseudocode The following is deliberately simplified: implement a validated bootstrap/difference estimation and explicit sample policy for the real deployment. \\\python def classify benchmark(baseline, candidate, policy): if baseline.workload hash != candidate.workload hash: return "INVALID WORKLOAD DRIFT" if baseline.hardware fingerprint != candidate.hardware fingerprint: return "INVALID ENVIRONMENT DRIFT" if not baseline.telemetry complete or not candidate.telemetry complete: return "INVALID MEASUREMENT" if candidate.actual offered rps < policy.min arrival delivery: return "INVALID LOAD GENERATOR" if candidate.valid request count < policy.min request count: return "INCONCLUSIVE SMALL SAMPLE" Test the known good control on the same lane. if not candidate.reference control stable: return "INVESTIGATE BENCHMARK LANE" if candidate.critical quality failure: return "BLOCK RELEASE" Estimated candidate baseline change with an appropriate interval: lo, hi = candidate.p95 ttft delta ci ms if lo policy.material p95 regression ms: return "CONFIRMED REGRESSION" if hi <= policy.material p95 regression ms: return "NO MATERIAL REGRESSION DETECTED" return "INCONCLUSIVE RETEST" \\\ Important: a confidence interval for the p95 difference must be estimated with a method appropriate for correlated request level data and independent runs. The logic above concerns a material change relative to the baseline ; an independent absolute SLO gate must also run. Never auto promote solely because a relative regression threshold is not breached. 13. Production incident: false alarm blocks every release Scenario: A GPU inference team deploys no new serving code, yet nightly benchmarks suddenly report +22% p95 TTFT . A hardwired threshold fails every PR. Engineers attempt to “fix” the serving scheduler. Investigation: 1. Compare full manifests: the test runner moved to another GPU node; a driver update is present. 2. Reference control measurements also slowed on this node, despite unchanged runtime/model code. 3. A new background data preprocessing job is sharing CPU and saturating local I/O during model warmup. 4. The load generator underdelivers target arrivals on some runs, changing queueing comparisons. 5. An open loop benchmark with a pinned isolated runner and matched warmup shows candidate and baseline within the historical variance. Root cause: environment and load generator drift, not evidence of a candidate regression. Correct action: classify tests INVALID/INCONCLUSIVE, isolate runner resources and restore a pinned reference lane, then rerun candidate versus control. Keep release blocked until valid evidence exists; do not bypass gates or change production solely to silence an alert. Prevention: immutable hardware/software/workload fingerprints, a scheduled known good control run, generator health gates, multi run comparisons and benchmark lane incident ownership. 14. A different incident: false negative masks real long context regression Scenario: All tests report stable aggregate p95 TTFT, but customers processing 64k token documents complain about long first token delays. Root cause: the evaluation dataset's share of long context prompts dropped from 12% to 2%, hiding the affected workload in aggregate metrics. New cache warmth made the fixed prompt list seem faster. Diagnosis and fix: compare length histograms and sample quotas; restore the representative long context stratum; run warm and cold prefix experiments; require slice level SLO gates and minimum sample sizes. This is workload drift causing a false negative , not just noisy test hardware. Prevention: fixture distribution fingerprints, mandatory critical cohorts, and per slice regression results. 15. Monitoring, reporting and common mistakes Every CI benchmark result should show: Exact experiment/configuration/workload fingerprint and environment comparison status. Baseline and candidate offered rate, achieved arrivals, completed/errored original requests and valid samples. Warmup and measurement windows, cache state and generator schedule lag. Per run/per slice TTFT/ITL/TPOT percentiles, error rate, joint SLO goodput and unit costs. The size and confidence interval of candidate minus baseline , practical threshold, and hard SLO status. Reference control drift and detected changes in hardware/driver/dataset or metric definitions. A reason coded PASS / FAIL / INCONCLUSIVE / INVALID decision with reproduction instructions and links to raw artifacts. Avoid these errors: Blocking release after a single percent change just above threshold without checking measurement quality. Silently passing an evaluation because a required GPU test did not run. Ignoring warmup or accidentally benchmarking one warm and one cold cache. Treating individual generated tokens as independent latency samples. Averaging worker level p95/p99 into a global percentile. Retuning the regression threshold after seeing results without recording the policy revision. Using “no statistically significant difference” to assert equivalence. Comparing unlike GPUs or changed prompt mixes as though they were identical. Automatically replacing an immutable reference baseline after every small change. Hiding errors and timeouts by benchmarking only successful streams. 16. Interview ready answer “I'd fingerprint the model, engine, GPU, workload, cache state and load generator for every inference benchmark, and run a known good control on the same lane. I'd separate hardware/workload/measurement drift from candidate behavior, use randomized or paired repeated tests, and compare p95 TTFT/ITL and joint SLO goodput with both hard limits and prespecified material regression margins. A single noisy run or insignificant p value isn't enough. If conditions aren't comparable or uncertainty overlaps harmful outcomes, I'd mark the gate INCONCLUSIVE, fix the lane and rerun rather than automatically passing or rolling back.” Authoritative research references Microsoft Research — Reproducible LLM Inference Benchmarking (October 2026) vLLM — Continuous Benchmarking Performance Dashboard NVIDIA — AIPerf Benchmarking at Scale (September 2026) NVIDIA — AIPerf Load Generator Options NVIDIA — AIPerf Warmup Configuration NVIDIA — AIPerf Metrics Reference
How do you design statistically reliable inference performance experiments using paired runs, confidence intervals, and sequential testing?
Designing Statistically Reliable LLM Inference Performance Experiments Statistically reliable LLM inference experiments compare serving configurations with enough control, replication and explicit decision rules to distinguish a reproducible operational improvement from random variation. Paired designs reduce unwanted differences; confidence intervals quantify uncertainty; and valid sequential methods permit a team to stop when evidence is sufficiently strong without using an invalid “peek until p < 0.05” strategy. A result like “candidate B is 6% faster” is incomplete unless you can answer which metric, on what request distribution, at what load, with how many independent repetitions, using what uncertainty calculation, and with which predeclared deployment gate . Curriculum positioning: Question 258 teaches representative load tests, Question 266 investigates actual regressions, Question 281 builds a CI evaluation workflow and Question 282 explains environment/workload drift and false alerts. Here we formalize experiment design, estimands, paired randomization, bootstrap/permutation inference, sample size and power planning, multiple testing and safe sequential monitoring. 1. Specify the estimand—not just “latency” An estimand is the exact outcome you want to compare between candidate B and control A. Good examples: Difference in p95 client observed time to first token (TTFT) at a fixed offered arrival rate and prompt/output token mix. Difference in p95 per request time per output token (TPOT) during the same warm cache steady state protocol. Difference in original request SLO compliant goodput at peak load, including failed and timed out user requests in the denominator. Ratio of total GPU cost per successfully completed, quality acceptable SLO original request under identical price assumptions. Probability that candidate B violates a latency ceiling on an important long context slice . Be precise: A p95 difference is \p95(B) − p95(A)\, not \p95(B−A)\. These are different mathematical quantities. A p95 pooled over all requests differs from the mean of run level p95 values . vLLM defines client observed TTFT from request submission until first streamed output. Its ITL values are gaps between streamed outputs; its TPOT is calculated per request using total output tokens and decode duration. Under speculative decoding, one streamed chunk can contain multiple tokens, so ITL and TPOT can diverge. Never compute a latency percentile only from successful requests while silently dropping errors/timeouts. Report both latency of valid responses and SLO goodput over every original offered request . Reference: https://docs.vllm.ai/en/latest/benchmarking/cli/ 2. Decide the independent experimental unit Inference data are usually hierarchical: A trial/run occurs on a particular GPU node, runtime state and time window. A run contains many user requests . Each request contains many streamed tokens or output chunks . Requests in the same run share GPU queues, CUDA state, scheduling and network conditions. Tokens are not thousands of independent latency experiments . Treating 1 million output tokens from one run as 1 million independent experimental replicates produces unrealistically narrow confidence intervals. The right unit depends on the claim: Engine/configuration impact under repeated deployments: use independently repeated run pairs or separately randomized cluster/time blocks. Request level paired answer quality: the same task/prompt is a useful paired unit, with appropriate repeated decoding or judge variance handling. Online user level experiment: randomize at an independent, interference safe unit, such as account, session or an appropriately isolated serving cell; naive per request randomization may create interference through shared GPU capacity. Single fixed trace replay: repeated traces help isolate software behavior but do not establish generalization to every production workload. Use cluster aware or block aware inference when failures/load are correlated within GPU nodes or experiment sessions. 3. Paired A/B runs: why matching helps Instead of comparing a random baseline Tuesday with a candidate Friday, create comparable blocks : 1. Same approved GPU SKU/topology and serving engine dependencies except the intentional candidate change. 2. Same immutable prompt/output distribution, arrival schedule, target QPS, generation settings and cache protocol. 3. Same benchmark client capacity and a healthy isolated GPU lane. 4. Randomized or counterbalanced order to reduce time trends—such as \ABBA\, \BAAB\ or randomly selected A/B order per block. 5. Repeat across several independent blocks, restoring the documented warmup and cache state before each. Paired differences \d i = B i − A i\ remove some run to run environmental variability. This is useful when both versions experience similar time of day or GPU clock effects. But pairing is valid only when the pairs have a meaningful connection. Two unrelated benchmark samples are not paired just because their row numbers match. Pairing can also be damaged by A contaminating B's cache or thermal state; use explicit reset/warmup and randomize execution order. 4. Beginner friendly numerical example: estimate the mean paired slowdown Suppose you conduct 8 independent matched run pairs at the same offered demand. The statistic for each run is its median TTFT in milliseconds , and B is the proposed replacement. Illustrative run level median TTFT : | Pair | Baseline A (ms) | Candidate B (ms) | B − A (ms) | | | :| :| :| | 1 | 190 | 194 | +4 | | 2 | 195 | 200 | +5 | | 3 | 188 | 191 | +3 | | 4 | 192 | 198 | +6 | | 5 | 197 | 201 | +4 | | 6 | 191 | 196 | +5 | | 7 | 193 | 198 | +5 | | 8 | 189 | 193 | +4 | Paired differences: [4, 5, 3, 6, 4, 5, 5, 4] ms . Mean paired change: \36/8 = +4.5 ms\. Sample standard deviation of differences: approximately 0.926 ms . Standard error: \0.926 / sqrt(8) ≈ 0.327 ms\. With a two sided 95% Student t interval and seven degrees of freedom, \t≈2.365\. Interval for the mean of run level median differences : \4.5 ± 2.365×0.327 ≈ [3.73, 5.27] ms\. Interpretation: B appears consistently about 4.5 ms slower in this controlled example . The interval concerns the mean paired difference in run level medians , not the global p95 TTFT difference. Suppose the predetermined practical regression tolerance is +10 ms , and absolute customer p95 SLOs pass. Because the interval's upper endpoint is 5.27 ms , B's degradation is plausibly smaller than this operational threshold. That does not mean B is identical to A; a small slowdown can be real yet acceptable. The t method assumes independent run pair differences with suitable distributional behavior. With only eight pairs, inspect outliers and skew and consider a robust sensitivity analysis. All numbers are synthetic teaching data. 5. Statistical significance versus practical significance Before collecting results, set: \delta\: maximum acceptable regression magnitude , such as +10 ms mean paired median TTFT change or +25 ms pooled p95 change. \alpha\: the allowed false alarm probability for a prespecified statistical test. \power\: desired probability of detecting a practically important effect under a planning model. \minimum sample size\ and \maximum runtime/budget\. Absolute SLO ceilings, critical cohort gates and mandatory quality/reliability requirements. Two different decisions: Detect any change: test \H0: effect=0\; it may flag an irrelevant 1 ms slowdown given enormous data. Establish noninferiority: show the upper confidence bound on candidate − baseline latency lies below a prespecified +delta margin. An ordinary non significant difference does not establish noninferiority. Establish improvement: show candidate is faster than the baseline by enough to justify operational risk, using a prespecified confidence bound or decision threshold. A candidate can be statistically slower but practically acceptable , or apparently faster while the interval is too wide to justify promotion. Always apply independent hard user facing SLO and safety gates . 6. Bootstrap inference: resample at the correct level When p95/p99 statistics are skewed and analytic standard errors are awkward, bootstrap intervals can help, but only if you respect the experimental design. Unpaired case: resample independent A runs and independent B runs, preserving the difference in estimands. Do not automatically resample individual tokens. Paired run case: sample whole run pair indices with replacement, recompute the chosen difference for each sample, and use a documented interval construction such as percentile or BCa where its assumptions and numerical stability are suitable. Hierarchical case: use a block/cluster bootstrap over run pairs, then optionally within run request traces, with a design that preserves time clustering and offered load structure. Simply sampling every request independently can underestimate GPU queue correlation. SciPy's \scipy.stats.bootstrap\ supports \paired=True\ and \method="BCa"\. Its paired option resamples the same index positions across arrays , preserving correspondence. BCa can fail or be unstable when data are degenerate or sample size tiny; examine diagnostics. Illustrative Python code for paired run level mean differences: \\\python import numpy as np from scipy.stats import bootstrap baseline = np.array([190, 195, 188, 192, 197, 191, 193, 189]) candidate = np.array([194, 200, 191, 198, 201, 196, 198, 193]) Each element is the median TTFT from an independent paired run. def mean paired delta(a, b, axis= 1): return np.mean(b a, axis=axis) ci = bootstrap( (baseline, candidate), mean paired delta, paired=True, vectorized=True, n resamples=9999, confidence level=0.95, method="BCa", rng=np.random.default rng(42), ).confidence interval print("point estimate:", mean paired delta(baseline, candidate)) print("95% CI:", ci.low, ci.high) \\\ Limits: the calculation assumes reasonably independent run pairs. For a difference in pooled p95 , implement a statistic that reconstructs per request data in each resampled run, then recomputes each pooled percentile; do not replace it with the mean of per run p95s. Reference: https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.bootstrap.html 7. Paired permutation tests: when and how A paired randomization or sign flip permutation test asks whether the observed paired statistic is surprising under the chosen null and exchangeability assumptions. In SciPy, \permutation type="samples"\ swaps labels within pairs, whereas \permutation type="pairings"\ reshuffles which items are matched. These test different hypotheses . When comparing baseline/candidate measured on paired runs, swapping conditions within pairs (or sign flipping their differences under the correct symmetry/randomization assumptions) is usually the relevant concept. Example: \\\python import numpy as np from scipy.stats import permutation test a = np.array([190, 195, 188, 192, 197, 191, 193, 189]) b = np.array([194, 200, 191, 198, 201, 196, 198, 193]) def mean difference(x, y, axis= 1): return np.mean(y x, axis=axis) result = permutation test( (a, b), mean difference, permutation type="samples", alternative="greater", n resamples=9999, rng=np.random.default rng(42), ) print(result.statistic, result.pvalue) \\\ Here \alternative="greater"\ tests evidence that candidate B is slower under the chosen statistic. This is a fixed sample paired test; it is not a sequentially valid test if rerun after every additional pair until p < 0.05. For eight pairs, at most \2 8=256\ unique within pair swaps exist, so sufficiently many requested resamples trigger exact enumeration in SciPy. Reference: https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.permutation test.html 8. Why “peek until significant” produces false positives Suppose an engineering pipeline runs 100 benchmarks and repeatedly tests after 10, 20, 30 … completed pairs with an ordinary fixed sample p < 0.05 threshold. If it stops the first time significance appears, the overall false positive risk can exceed 5%. The ordinary p value assumes the planned testing/stopping structure; the repeated unadjusted looks change that structure. Wrong rule: “Run until the first confidence interval excludes zero, then publish an improvement.” Valid families of strategies: 1. Fixed horizon: predeclare 20 independent run pairs, analyze only after collecting them; monitor health separately and stop early for safety if necessary. 2. Group sequential design: predeclare a handful of interim analyses and use an alpha spending or group sequential boundary (e.g., O'Brien–Fleming style) that controls overall type I error across planned looks. 3. Confidence sequences: use a mathematically time uniform interval procedure with the required boundedness/moment or martingale assumptions; valid monitoring may stop at data dependent times within that procedure's guarantee. 4. Always valid/e process based testing: when assumptions are satisfied, use a predeclared nonnegative e process or other appropriate always valid test with controlled optional stopping. Do not call an ordinary bootstrap confidence interval a confidence sequence. The methods have different coverage guarantees. The confidence sequence literature (Howard, Ramdas, McAuliffe, Sekhon) establishes time uniform uncertainty bounds under explicit statistical conditions, not a magic correction for invalid benchmark data or GPU interference. Reference: https://arxiv.org/abs/1810.08240 9. Simple sequential decision example Hypothetical experiment: candidate B is proposed to improve p95 TTFT. Baseline p95 is 900 ms . The company requires B to be at least 30 ms faster to justify rollout risk while maintaining quality and reliability. Planned evidence collection: Five independent A/B experiment blocks at stage 1, then five more blocks at stages 2 and 3. The engine tests the same offered workload with randomized A/B order per block. A prespecified group sequential method assigns conservative early boundaries and a final boundary; tests follow that method's documented critical values. The measured difference is \delta = p95(B) p95(A)\, so desired improvement means \delta <= 30 ms\. Illustrative reports: At 5 blocks: estimate \ 48 ms\, but uncertainty includes \ 10 ms\. Continue ; the evidence does not establish the minimum desired 30 ms effect. At 10 blocks: estimate \ 42 ms\, uncertainty includes \ 29 ms\. Continue ; near boundary. At 15 blocks: estimate \ 41 ms\, and a method adjusted upper bound is \ 33 ms\. Potential PASS , provided the prespecified sequential boundary, other hard SLO/quality gates and experiment comparability checks also pass. Caution: these bounds are illustrative inputs , not calculated boundaries from a named alpha spending function. The actual implementation must compute and validate stage specific bounds from a selected, tested method. An ordinary fixed 95% interval checked three times would not justify this pass. 10. Sample size and statistical power Aim: detect a performance change that matters operationally, with sufficiently small chance of missing it. For a rough fixed horizon paired mean difference planning calculation, assume independent approximately normal run pair differences with sample standard deviation \sigma d\, two sided type I error \alpha\, target effect \delta\, and desired power \1 beta\: \\\text n pairs ≈ (z (1−alpha/2) + z (1−beta))² × sigma d² / delta² \\\ Example: historical paired run difference SD = 12 ms , meaningful change = 8 ms , alpha = 0.05, desired power = 80%. \z (0.975) ≈ 1.96\, \z (0.8) ≈ 0.84\. ≈ (1.96 + 0.84)^2 × 12^2 / 8^2\. ≈ 7.84 × 144 / 64 ≈ 17.64\. Plan at least 18 independent matched pairs by this normal approximation calculation; a small sample t calculation or added variance/attrition buffer may require more. Limitations: this formula is for a mean paired difference , not p95/p99 tail quantiles, a bounded success fraction, time correlated workload traces, or an anytime valid sequential test. For tail latency, use pilot data and a simulation based power analysis that preserves the observed latency distribution and block level dependence. If the budget allows only 4 experiment pairs and the detectable effect is much larger than the team's desired 5 ms improvement, report INCONCLUSIVE instead of claiming no regression. 11. Multiple comparisons across many deployment knobs LLM inference engineers often test: 5 GPU SKUs, 3 quantization/KV cache settings, 4 scheduler configurations, p50/p95/p99 TTFT, TPOT and end to end latency, 6 tenant/context length slices. If each statistical hypothesis is tested at 5%, uncorrected repeated comparisons create many opportunities for false discoveries. Strategies: Primary metric and gate first: one or a small predeclared set of decision driving metrics; use others diagnostically. Holm–Bonferroni control: controls family wise error rate for a chosen family of hypotheses when applicable. Benjamini–Hochberg false discovery rate: may suit exploratory bulk screening but FDR is not a substitute for strict safety or SLO gates . Hierarchical testing: test global performance first, then predeclared critical cohorts under an appropriate procedure. Holdout confirmation: use one evaluation set for candidate exploration and a separate untouched confirmation set for the selected winner. Sequential correction still required: multiplicity across both hypotheses and interim looks must be considered in the same inference design. Avoid “try every configuration, then report the winning p value from the same runs” as if only one hypothesis had been tested. 12. Confidence intervals for SLO goodput and reliability A release might have: Baseline: 9,900 of 10,000 original requests satisfy completion and latency objectives (99.0%). Candidate: 9,940 of 10,000 satisfy them (99.4%). Point estimate: candidate improves 0.4 percentage points . But: Request outcomes may be correlated by GPU lane, failure episode, prompt stratum or time of day. Paired original requests must share a meaningful correspondence; an online A/B where different users receive different traffic is not a naturally paired outcome. The 10,000 requests in one one hour run do not represent 10,000 independent infrastructure deployments. Estimate the confidence interval for the difference in success rates , using cluster aware methods or appropriate paired binomial methods where the design warrants them . Separately ensure hard quality, security and critical tenant floors pass. A 0.4 point aggregate success improvement cannot justify a new safety failure. Joint SLO compliance means the same original request meets every mandatory latency/completion requirement. Separate p95 TTFT and TPOT reports alone do not establish a joint completion fraction. 13. Production incident: a false winner from repeated peeking Scenario: a team compares ten new vLLM scheduler/KV configurations. A CI script executes a short five minute benchmark for each, computes unadjusted p values at every additional 1,000 requests, and stops the first run showing p < 0.05. The selected candidate appears 9% faster . Under a representative long context peak test it is actually 12% slower , and user facing p95 TTFT breaches its SLO. Root causes: 1. Repeated optional stopping inflated the false discovery risk. 2. Selection across ten candidates multiplied the testing problem. 3. A short warm cache test lacked long context and bursty arrival cases. 4. Treating all token observations as independent made intervals too narrow. 5. No independent holdout confirmation was performed. Corrective action: Restore the approved serving configuration; protect production SLOs. Predeclare a primary p95 TTFT estimand and minimum practical effect. Block or randomize comparable A/B GPU runs, preserving independent run units. Reserve separate candidate selection and confirmation experiments. Use fixed horizon or predeclared group sequential inference—not naive repeated fixed sample testing. Report per slice request success, TTFT/TPOT, quality, cost and clearly reason coded decision evidence. Prevention: an experiment registry and CI validator that rejects unregistered metrics, changed thresholds, unsupported interim looks and missing run level artifacts. 14. Recommended engineering workflow 1. Register a hypothesis: candidate version, baseline, workload, primary metric, meaningful improvement/regression threshold, alpha/power, maximum compute budget and protected SLOs. 2. Pilot for variance: collect repeated control runs to estimate relevant run pair variability and the expected noise floor. 3. Choose a design: paired randomized A/B blocks, fixed or sequential monitoring, and a valid unit of randomization. 4. Determine sample budget: use planning formulas for means or simulation for p95, reliability and heavy tailed latency. 5. Freeze comparability: model/container/GPU hashes, cache protocol, arrivals, prompt/output mix, instrumentation and generator capacity. 6. Run with monitoring: continuously abort for dangerous SLO/safety failures; such aborts are health measures, not automatic statistical acceptance. 7. Estimate effect and uncertainty: paired/bootstrap/permutation/group sequential analysis as prespecified. 8. Account for multiplicity: all candidate variants, primary metrics and interim looks under the chosen plan. 9. Require confirmation: protected production quality slices, failure cases and a holdout or independent matched test. 10. Save evidence: immutable experiment ID, raw request traces, run pair results, model/engine fingerprint, alpha budget, effect estimate, interval and decision. 11. Gate release: \PASS\ only when comparator validity and all hard SLO/quality/reliability constraints hold; \FAIL\ on confirmed breach; \INCONCLUSIVE\ on weak/missing evidence. 12. Promote through guarded rollout using the separate deployment practices from Question 264. 15. Common mistakes Treating millions of streamed tokens from one node as millions of independent experiments. Calling two unaligned request batches “paired” merely because both contain the same number of requests. Comparing p95 of differences with the difference of p95s . Concluding equivalence because \p 0.05\. Looking every minute at ordinary 95% CIs and stopping as soon as one excludes zero. Using a standard paired permutation test while continuously adding pairs without a stopping correction. Using the bootstrap with \paired=False\ for genuine matched runs. Reusing the same test data to choose and “confirm” the winning configuration. Reporting aggregate p95 improvement while a critical long context slice regresses. Failing to include timeouts, errors and aborted streams in SLO goodput. Claiming a simulation derived interval is universally valid without checking noise, dependence, sample size and assumptions. Hiding a practical meaningfulness threshold behind a generic “significant” label. 16. Interview ready answer “I'd predeclare the inference metric and meaningful regression margin, then collect comparable baseline/candidate runs in randomized matched blocks. The independent unit is usually a run or workload block, not every token. I'd estimate effects with paired differences and cluster aware confidence intervals, and use a valid fixed sample or sequential testing plan—never ordinary p < 0.05 peeking. I'd plan enough repetitions for the desired effect size, correct for many candidate comparisons, and confirm the selected winner on a holdout workload. Deployment is allowed only after quality, client observed latency, joint SLO goodput and reliability gates pass.” Research references SciPy — Bootstrap Confidence Intervals and Paired Resampling SciPy — Paired Permutation Tests Howard et al. — Time Uniform Confidence Sequences vLLM — Serving Benchmark Metrics and Goodput vLLM — TTFT, ITL and TPOT Metric Definitions