How do you design statistically reliable inference performance experiments using paired runs, confidence intervals, and sequential testing?

# Designing Statistically Reliable LLM Inference Performance Experiments

Statistically reliable LLM inference experiments compare serving configurations with enough control, replication and explicit decision rules to distinguish a reproducible operational improvement from random variation. Paired designs reduce unwanted differences; confidence intervals quantify uncertainty; and *valid sequential methods* permit a team to stop when evidence is sufficiently strong without using an invalid “peek until p < 0.05” strategy.

A result like “candidate B is 6% faster” is incomplete unless you can answer which metric, on what request distribution, at what load, with how many independent repetitions, using what uncertainty calculation, and with which predeclared deployment gate.

Curriculum positioning: Question 258 teaches representative load tests, Question 266 investigates actual regressions, Question 281 builds a CI evaluation workflow and Question 282 explains environment/workload drift and false alerts. Here we formalize experiment design, estimands, paired randomization, bootstrap/permutation inference, sample-size and power planning, multiple testing and safe sequential monitoring.

1. Specify the estimand—not just “latency”

An estimand is the exact outcome you want to compare between candidate B and control A.

Good examples:

• Difference in p95 client-observed time to first token (TTFT) at a fixed offered arrival-rate and prompt/output-token mix.
• Difference in p95 per-request time per output token (TPOT) during the same warm-cache steady-state protocol.
• Difference in original-request SLO-compliant goodput at peak load, including failed and timed-out user requests in the denominator.
• Ratio of total GPU cost per successfully completed, quality-acceptable SLO original request under identical price assumptions.
• Probability that candidate B violates a latency ceiling on an important long-context slice.

Be precise:

• A p95 difference is \p95(B) − p95(A)\, not \p95(B−A)\. These are different mathematical quantities.
• A p95 pooled over all requests differs from the mean of run-level p95 values.
• vLLM defines client-observed TTFT from request submission until first streamed output. Its ITL values are gaps between streamed outputs; its TPOT is calculated per request using total output tokens and decode duration. Under speculative decoding, one streamed chunk can contain multiple tokens, so ITL and TPOT can diverge.
• Never compute a latency percentile only from successful requests while silently dropping errors/timeouts. Report both latency of valid responses and SLO-goodput over every original offered request.

Reference: https://docs.vllm.ai/en/latest/benchmarking/cli/

2. Decide the independent experimental unit

Inference data are usually hierarchical:

• A trial/run occurs on a particular GPU node, runtime state and time window.
• A run contains many user requests.
• Each request contains many streamed tokens or output chunks.
• Requests in the same run share GPU queues, CUDA state, scheduling and network conditions.

Tokens are not thousands of independent latency experiments. Treating 1 million output tokens from one run as 1 million independent experimental replicates produces unrealistically narrow confidence intervals.

The right unit depends on the claim:

• Engine/configuration impact under repeated deployments: use independently repeated *run pairs* or separately randomized cluster/time blocks.
• Request-level paired answer quality: the same task/prompt is a useful paired unit, with appropriate repeated decoding or judge variance handling.
• Online user-level experiment: randomize at an independent, interference-safe unit, such as account, session or an appropriately isolated serving cell; naive per-request randomization may create interference through shared GPU capacity.
• Single fixed trace replay: repeated traces help isolate software behavior but do not establish generalization to every production workload.

Use cluster-aware or block-aware inference when failures/load are correlated within GPU nodes or experiment sessions.

3. Paired A/B runs: why matching helps

Instead of comparing a random baseline Tuesday with a candidate Friday, create comparable blocks:

1. Same approved GPU SKU/topology and serving engine dependencies except the intentional candidate change.
2. Same immutable prompt/output distribution, arrival schedule, target QPS, generation settings and cache protocol.
3. Same benchmark-client capacity and a healthy isolated GPU lane.
4. Randomized or counterbalanced order to reduce time trends—such as \ABBA\, \BAAB\ or randomly selected A/B order per block.
5. Repeat across several independent blocks, restoring the documented warmup and cache state before each.

Paired differences \d_i = B_i − A_i\ remove some run-to-run environmental variability. This is useful when both versions experience similar time-of-day or GPU clock effects.

But pairing is valid only when the pairs have a meaningful connection. Two unrelated benchmark samples are not paired just because their row numbers match. Pairing can also be damaged by A contaminating B's cache or thermal state; use explicit reset/warmup and randomize execution order.

4. Beginner-friendly numerical example: estimate the mean paired slowdown

Suppose you conduct 8 independent matched run pairs at the same offered demand. The statistic for each run is its *median TTFT in milliseconds*, and B is the proposed replacement.

Illustrative run-level median TTFT:

| Pair | Baseline A (ms) | Candidate B (ms) | B − A (ms) | |---|---:|---:|---:| | 1 | 190 | 194 | +4 | | 2 | 195 | 200 | +5 | | 3 | 188 | 191 | +3 | | 4 | 192 | 198 | +6 | | 5 | 197 | 201 | +4 | | 6 | 191 | 196 | +5 | | 7 | 193 | 198 | +5 | | 8 | 189 | 193 | +4 |

• Paired differences: [4, 5, 3, 6, 4, 5, 5, 4] ms.
• Mean paired change: \36/8 = +4.5 ms\.
• Sample standard deviation of differences: approximately 0.926 ms.
• Standard error: \0.926 / sqrt(8) ≈ 0.327 ms\.
• With a two-sided 95% Student t interval and seven degrees of freedom, \t≈2.365\.
• Interval for the mean of run-level median differences: \4.5 ± 2.365×0.327 ≈ [3.73, 5.27] ms\.

Interpretation: B appears consistently about 4.5 ms slower in this controlled example. The interval concerns the mean paired difference in run-level medians, *not* the global p95 TTFT difference.

Suppose the predetermined practical regression tolerance is +10 ms, and absolute customer p95 SLOs pass. Because the interval's upper endpoint is 5.27 ms, B's degradation is plausibly smaller than this operational threshold. That does not mean B is identical to A; a small slowdown can be real yet acceptable.

The t method assumes independent run-pair differences with suitable distributional behavior. With only eight pairs, inspect outliers and skew and consider a robust sensitivity analysis. All numbers are synthetic teaching data.

5. Statistical significance versus practical significance

Before collecting results, set:

• \delta\: maximum acceptable regression magnitude, such as +10 ms mean paired median-TTFT change or +25 ms pooled p95 change.
• \alpha\: the allowed false-alarm probability for a prespecified statistical test.
• \power\: desired probability of detecting a practically important effect under a planning model.
• \minimum sample size\ and \maximum runtime/budget\.
• Absolute SLO ceilings, critical cohort gates and mandatory quality/reliability requirements.

Two different decisions:

• Detect any change: test \H0: effect=0\; it may flag an irrelevant 1-ms slowdown given enormous data.
• Establish noninferiority: show the upper confidence bound on candidate − baseline latency lies below a prespecified +delta margin. An ordinary non-significant difference does *not* establish noninferiority.
• Establish improvement: show candidate is faster than the baseline by enough to justify operational risk, using a prespecified confidence bound or decision threshold.

A candidate can be statistically slower but practically acceptable, or apparently faster while the interval is too wide to justify promotion. Always apply independent hard user-facing SLO and safety gates.

6. Bootstrap inference: resample at the correct level

When p95/p99 statistics are skewed and analytic standard errors are awkward, bootstrap intervals can help, but only if you respect the experimental design.

Unpaired case: resample independent A runs and independent B runs, preserving the difference in estimands. Do not automatically resample individual tokens.

Paired run case: sample whole run-pair indices with replacement, recompute the chosen difference for each sample, and use a documented interval construction such as percentile or BCa where its assumptions and numerical stability are suitable.

Hierarchical case: use a block/cluster bootstrap over run pairs, then optionally within-run request traces, with a design that preserves time clustering and offered-load structure. Simply sampling every request independently can underestimate GPU queue correlation.

SciPy's \scipy.stats.bootstrap\ supports \paired=True\ and \method="BCa"\. Its paired option resamples *the same index positions across arrays*, preserving correspondence. BCa can fail or be unstable when data are degenerate or sample size tiny; examine diagnostics.

Illustrative Python code for paired run-level mean differences: \\\`python import numpy as np from scipy.stats import bootstrap

baseline = np.array([190, 195, 188, 192, 197, 191, 193, 189]) candidate = np.array([194, 200, 191, 198, 201, 196, 198, 193])

# Each element is the median TTFT from an independent paired run. def mean_paired_delta(a, b, axis=-1): return np.mean(b - a, axis=axis)

ci = bootstrap( (baseline, candidate), mean_paired_delta, paired=True, vectorized=True, n_resamples=9999, confidence_level=0.95, method="BCa", rng=np.random.default_rng(42), ).confidence_interval

print("point estimate:", mean_paired_delta(baseline, candidate)) print("95% CI:", ci.low, ci.high) \\\`

Limits: the calculation assumes reasonably independent run pairs. For a difference in pooled p95, implement a statistic that reconstructs per-request data in each resampled run, then recomputes each pooled percentile; do not replace it with the mean of per-run p95s.

Reference: https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.bootstrap.html

7. Paired permutation tests: when and how

A paired randomization or sign-flip permutation test asks whether the observed paired statistic is surprising under the chosen null and exchangeability assumptions.

In SciPy, \`permutation_type="samples"\` swaps labels within pairs, whereas \permutation_type="pairings"\ reshuffles which items are matched. These test different hypotheses. When comparing baseline/candidate measured on paired runs, swapping conditions within pairs (or sign-flipping their differences under the correct symmetry/randomization assumptions) is usually the relevant concept.

Example: \\\`python import numpy as np from scipy.stats import permutation_test

a = np.array([190, 195, 188, 192, 197, 191, 193, 189]) b = np.array([194, 200, 191, 198, 201, 196, 198, 193])

def mean_difference(x, y, axis=-1): return np.mean(y - x, axis=axis)

result = permutation_test( (a, b), mean_difference, permutation_type="samples", alternative="greater", n_resamples=9999, rng=np.random.default_rng(42), )

print(result.statistic, result.pvalue) \\\`

Here \alternative="greater"\ tests evidence that candidate B is slower under the chosen statistic. This is a fixed-sample paired test; it is *not* a sequentially valid test if rerun after every additional pair until p < 0.05.

For eight pairs, at most \2**8=256\ unique within-pair swaps exist, so sufficiently many requested resamples trigger exact enumeration in SciPy.

Reference: https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.permutation_test.html

8. Why “peek until significant” produces false positives

Suppose an engineering pipeline runs 100 benchmarks and repeatedly tests after 10, 20, 30 … completed pairs with an ordinary fixed-sample p < 0.05 threshold. If it stops the first time significance appears, the overall false-positive risk can exceed 5%. The ordinary p-value assumes the planned testing/stopping structure; the repeated unadjusted looks change that structure.

Wrong rule: “Run until the first confidence interval excludes zero, then publish an improvement.”

Valid families of strategies:

1. Fixed horizon: predeclare 20 independent run pairs, analyze only after collecting them; monitor health separately and stop early for safety if necessary.
2. Group-sequential design: predeclare a handful of interim analyses and use an alpha-spending or group-sequential boundary (e.g., O'Brien–Fleming-style) that controls overall type-I error across planned looks.
3. Confidence sequences: use a mathematically time-uniform interval procedure with the required boundedness/moment or martingale assumptions; valid monitoring may stop at data-dependent times within that procedure's guarantee.
4. Always-valid/e-process-based testing: when assumptions are satisfied, use a predeclared nonnegative e-process or other appropriate always-valid test with controlled optional stopping.

Do not call an ordinary bootstrap confidence interval a confidence sequence. The methods have different coverage guarantees. The confidence-sequence literature (Howard, Ramdas, McAuliffe, Sekhon) establishes time-uniform uncertainty bounds under explicit statistical conditions, not a magic correction for invalid benchmark data or GPU interference.

Reference: https://arxiv.org/abs/1810.08240

9. Simple sequential decision example

Hypothetical experiment: candidate B is proposed to improve p95 TTFT. Baseline p95 is 900 ms. The company requires B to be at least 30 ms faster to justify rollout risk while maintaining quality and reliability.

Planned evidence collection:

• Five independent A/B experiment blocks at stage 1, then five more blocks at stages 2 and 3.
• The engine tests the same offered workload with randomized A/B order per block.
• A prespecified group-sequential method assigns conservative early boundaries and a final boundary; tests follow that method's documented critical values.
• The measured difference is \delta = p95(B)-p95(A)\, so desired improvement means \delta <= -30 ms\.

Illustrative reports:

• At 5 blocks: estimate \-48 ms\, but uncertainty includes \-10 ms\. Continue; the evidence does not establish the minimum desired -30 ms effect.
• At 10 blocks: estimate \-42 ms\, uncertainty includes \-29 ms\. Continue; near boundary.
• At 15 blocks: estimate \-41 ms\, and a method-adjusted upper bound is \-33 ms\. Potential PASS, provided the prespecified sequential boundary, other hard SLO/quality gates and experiment-comparability checks also pass.

Caution: these bounds are illustrative inputs, not calculated boundaries from a named alpha-spending function. The actual implementation must compute and validate stage-specific bounds from a selected, tested method. An ordinary fixed-95% interval checked three times would not justify this pass.

10. Sample size and statistical power

Aim: detect a performance change that matters operationally, with sufficiently small chance of missing it.

For a rough fixed-horizon paired mean difference planning calculation, assume independent approximately normal run-pair differences with sample standard deviation \sigma_d\, two-sided type-I error \alpha\, target effect \delta\, and desired power \1-beta\: \\\text n_pairs ≈ (z_(1−alpha/2) + z_(1−beta))² × sigma_d² / delta² \\\

Example: historical paired run-difference SD = 12 ms, meaningful change = 8 ms, alpha = 0.05, desired power = 80%.

• \z_(0.975) ≈ 1.96\, \z_(0.8) ≈ 0.84\.
• \n ≈ (1.96 + 0.84)^2 × 12^2 / 8^2\.
• \n ≈ 7.84 × 144 / 64 ≈ 17.64\.
• Plan at least 18 independent matched pairs by this normal-approximation calculation; a small-sample t calculation or added variance/attrition buffer may require more.

Limitations: this formula is for a mean paired difference, not p95/p99 tail quantiles, a bounded success fraction, time-correlated workload traces, or an anytime-valid sequential test. For tail latency, use pilot data and a simulation-based power analysis that preserves the observed latency distribution and block-level dependence.

If the budget allows only 4 experiment pairs and the detectable effect is much larger than the team's desired 5-ms improvement, report INCONCLUSIVE instead of claiming no regression.

11. Multiple comparisons across many deployment knobs

LLM inference engineers often test:

• 5 GPU SKUs,
• 3 quantization/KV-cache settings,
• 4 scheduler configurations,
• p50/p95/p99 TTFT, TPOT and end-to-end latency,
• 6 tenant/context-length slices.

If each statistical hypothesis is tested at 5%, uncorrected repeated comparisons create many opportunities for false discoveries.

Strategies:

• Primary metric and gate first: one or a small predeclared set of decision-driving metrics; use others diagnostically.
• Holm–Bonferroni control: controls family-wise error rate for a chosen family of hypotheses when applicable.
• Benjamini–Hochberg false discovery rate: may suit exploratory bulk screening but FDR is not a substitute for strict safety or SLO gates.
• Hierarchical testing: test global performance first, then predeclared critical cohorts under an appropriate procedure.
• Holdout confirmation: use one evaluation set for candidate exploration and a separate untouched confirmation set for the selected winner.
• Sequential correction still required: multiplicity across both *hypotheses and interim looks* must be considered in the same inference design.

Avoid “try every configuration, then report the winning p-value from the same runs” as if only one hypothesis had been tested.

12. Confidence intervals for SLO goodput and reliability

A release might have:

• Baseline: 9,900 of 10,000 original requests satisfy completion and latency objectives (99.0%).
• Candidate: 9,940 of 10,000 satisfy them (99.4%).
• Point estimate: candidate improves 0.4 percentage points.

But:

• Request outcomes may be correlated by GPU lane, failure episode, prompt stratum or time of day.
• Paired original requests must share a meaningful correspondence; an online A/B where different users receive different traffic is not a naturally paired outcome.
• The 10,000 requests in one one-hour run do not represent 10,000 independent infrastructure deployments.
• Estimate the confidence interval for the difference in success rates, using cluster-aware methods or appropriate paired-binomial methods *where the design warrants them*.
• Separately ensure hard quality, security and critical-tenant floors pass. A 0.4-point aggregate success improvement cannot justify a new safety failure.

Joint SLO compliance means the same original request meets every mandatory latency/completion requirement. Separate p95 TTFT and TPOT reports alone do not establish a joint completion fraction.

13. Production incident: a false winner from repeated peeking

Scenario: a team compares ten new vLLM scheduler/KV configurations. A CI script executes a short five-minute benchmark for each, computes unadjusted p-values at every additional 1,000 requests, and stops the first run showing p < 0.05. The selected candidate appears 9% faster. Under a representative long-context peak test it is actually 12% slower, and user-facing p95 TTFT breaches its SLO.

Root causes:

1. Repeated optional stopping inflated the false-discovery risk.
2. Selection across ten candidates multiplied the testing problem.
3. A short warm-cache test lacked long-context and bursty arrival cases.
4. Treating all token observations as independent made intervals too narrow.
5. No independent holdout confirmation was performed.

Corrective action:

• Restore the approved serving configuration; protect production SLOs.
• Predeclare a primary p95 TTFT estimand and minimum practical effect.
• Block or randomize comparable A/B GPU runs, preserving independent run units.
• Reserve separate candidate-selection and confirmation experiments.
• Use fixed-horizon or predeclared group-sequential inference—not naive repeated fixed-sample testing.
• Report per-slice request success, TTFT/TPOT, quality, cost and clearly reason-coded decision evidence.

Prevention: an experiment registry and CI validator that rejects unregistered metrics, changed thresholds, unsupported interim looks and missing run-level artifacts.

14. Recommended engineering workflow

1. Register a hypothesis: candidate version, baseline, workload, primary metric, meaningful improvement/regression threshold, alpha/power, maximum compute budget and protected SLOs.
2. Pilot for variance: collect repeated control runs to estimate relevant run-pair variability and the expected noise floor.
3. Choose a design: paired randomized A/B blocks, fixed or sequential monitoring, and a valid unit of randomization.
4. Determine sample budget: use planning formulas for means or simulation for p95, reliability and heavy-tailed latency.
5. Freeze comparability: model/container/GPU hashes, cache protocol, arrivals, prompt/output mix, instrumentation and generator capacity.
6. Run with monitoring: continuously abort for dangerous SLO/safety failures; such aborts are health measures, not automatic statistical acceptance.
7. Estimate effect and uncertainty: paired/bootstrap/permutation/group-sequential analysis as prespecified.
8. Account for multiplicity: all candidate variants, primary metrics and interim looks under the chosen plan.
9. Require confirmation: protected production-quality slices, failure cases and a holdout or independent matched test.
10. Save evidence: immutable experiment ID, raw request traces, run-pair results, model/engine fingerprint, alpha budget, effect estimate, interval and decision.
11. Gate release: \PASS\ only when comparator validity and all hard SLO/quality/reliability constraints hold; \FAIL\ on confirmed breach; \INCONCLUSIVE\ on weak/missing evidence.
12. Promote through guarded rollout using the separate deployment practices from Question 264.

15. Common mistakes

• Treating millions of streamed tokens from one node as millions of independent experiments.
• Calling two unaligned request batches “paired” merely because both contain the same number of requests.
• Comparing *p95 of differences* with the *difference of p95s*.
• Concluding equivalence because \p > 0.05\.
• Looking every minute at ordinary 95% CIs and stopping as soon as one excludes zero.
• Using a standard paired permutation test while continuously adding pairs without a stopping correction.
• Using the bootstrap with \paired=False\ for genuine matched runs.
• Reusing the same test data to choose and “confirm” the winning configuration.
• Reporting aggregate p95 improvement while a critical long-context slice regresses.
• Failing to include timeouts, errors and aborted streams in SLO-goodput.
• Claiming a simulation-derived interval is universally valid without checking noise, dependence, sample size and assumptions.
• Hiding a practical meaningfulness threshold behind a generic “significant” label.

16. Interview-ready answer

“I'd predeclare the inference metric and meaningful regression margin, then collect comparable baseline/candidate runs in randomized matched blocks. The independent unit is usually a run or workload block, not every token. I'd estimate effects with paired differences and cluster-aware confidence intervals, and use a valid fixed-sample or sequential testing plan—never ordinary p < 0.05 peeking. I'd plan enough repetitions for the desired effect size, correct for many candidate comparisons, and confirm the selected winner on a holdout workload. Deployment is allowed only after quality, client-observed latency, joint SLO-goodput and reliability gates pass.”

Research references

• SciPy — Bootstrap Confidence Intervals and Paired Resampling
• SciPy — Paired Permutation Tests
• Howard et al. — Time-Uniform Confidence Sequences
• vLLM — Serving Benchmark Metrics and Goodput
• vLLM — TTFT, ITL and TPOT Metric Definitions