How do you evaluate the accuracy and latency trade-offs of FP8 versus BF16 KV-cache quantization?

Evaluate FP8 KV-cache quantization against a BF16 baseline on the same model, workload, serving backend and hardware. FP8 reduces nominal KV tensor storage by about half, but performance depends on supported attention kernels, calibration scales and cache pressure; accuracy can degrade on specific long-context or reasoning tasks. Question 246 explained *how* cache bytes shrink; this lesson focuses on a rigorous go/no-go experiment.

1. Keep the experiment controlled

• Fix the model weights, tokenizer, prompt templates, generation settings, maximum context length, GPU type, tensor parallelism, batch/load profile and software versions.
• Change only KV-cache precision first: BF16 baseline vs FP8 with default scales. Then add calibrated FP8 as a third experiment.
• If backend selection also changes the attention computation (for example, FP8 attention quantizes Q as well), disclose that the experiment is FP8 KV + attention rather than KV precision alone.
• Benchmark cold/warm caches and low/peak concurrency separately.

2. What to measure

Quality:

• Task accuracy/pass@1, exact-match or retrieval correctness, long-context needle/retrieval tests, hallucination rate and output validity.
• Slice by prompt length (e.g., 2k, 8k, 32k, 128k), generation length, language and difficult reasoning tasks.
• Use several randomized generations for stochastic decoding; one mismatched response does not demonstrate quality regression.

Performance:

• GPU KV memory and cache capacity, actual KV transfer bytes if disaggregated, achievable concurrency, p50/p95 TTFT, p95 inter-token latency, generated tokens/s and SLO-compliant goodput.
• Include quantize/dequantize overhead, scales and metadata, preemption counts and hardware/kernel differences.

3. Beginner-friendly numerical example

Imagine evaluation on 1,000 labeled long-context questions plus a fixed high-load serving test:

BF16 KV cache

• 920 correct / 1,000 = 92.0% accuracy.
• p95 TTFT = 700 ms; p95 inter-token latency = 38 ms.
• Cache budget occupied = 40 GiB in a particular test.

Uncalibrated FP8 KV cache

• 880 correct / 1,000 = 88.0% accuracy.
• p95 TTFT = 650 ms; p95 ITL = 35 ms.
• Approximate same-token KV payload = 20 GiB, excluding scaling/metadata.

Calibrated FP8 KV cache

• 913 correct / 1,000 = 91.3% accuracy.
• p95 TTFT = 660 ms; p95 ITL = 36 ms.
• Approximate KV payload remains 20 GiB plus scale/metadata.

Calculate the trade-offs:

• Uncalibrated FP8 loses 4.0 accuracy percentage points relative to BF16, while TTFT improves 50/700 ≈ 7.1%.
• Calibrated FP8 loses 0.7 percentage points, while TTFT improves 40/700 ≈ 5.7%.
• If the release rule permits no more than 1.0 percentage point degradation *and* TTFT must not regress, calibrated FP8 passes this illustrative point-estimate screen; uncalibrated FP8 fails.
• Do not ship based solely on these 1,000 examples: evaluate uncertainty, important task slices, tail behavior and incident thresholds before rollout.

All figures are illustrative rather than claims about a specific model.

4. How calibration changes quality

• FP8 has less precision and dynamic range than BF16. Scale factors map tensor magnitudes into the representable interval.
• Per-tensor scaling shares a scale across a tensor; per-attention-head scaling uses finer granularity and may help when heads have different distributions.
• vLLM documents default scales of 1.0, dataset-based calibration with llm-compressor, and per-head scales where its supported backend allows them.
• Do not assume calibration always fully restores BF16 quality. Test model families, atypical activations, long prompts, sliding-window layers and supported selective BF16 fallbacks.

5. Latency and capacity do not scale linearly with bit width

• BF16 → FP8 cuts nominal bytes per KV element 2 → 1, but does not guarantee 2× requests/s or half the TTFT.
• Reduced KV memory may indirectly increase allowed concurrency and reduce preemptions, improving throughput.
• Some kernels operate directly on FP8; others dequantize KV before attention, adding compute. Cache-block packing and host/network format matter too.
• FP8 attention can change query precision as well as KV precision; results are backend-dependent.
• In distributed serving, verify that data crosses the network in compact FP8 format before claiming a bandwidth improvement.

6. Production rollout

1. Run repeatable offline quality suites and matched load tests.
2. Track long-context accuracy against BF16 by length bucket; explicitly test failure-prone slices.
3. Shadow test with real traffic and compare answers where evaluable.
4. Canary a small percentage of eligible requests, watching p95/p99 TTFT, ITL, throughput, errors and quality signals.
5. Roll back if a predeclared accuracy or latency guardrail fails; preserve BF16 as a safe fallback for unsupported or sensitive configurations.
6. Re-evaluate after changing models, hardware, attention backends, calibration dataset or context window.

Interview-ready answer: “I would compare BF16, uncalibrated FP8 and calibrated FP8 under identical model/hardware/load settings, tracking per-task and long-context accuracy plus p95 TTFT, ITL, concurrency and memory. FP8 approximately halves KV payload bytes but may affect quality and kernels. I'd ship only if the measured accuracy budget and latency SLOs hold under canary load.”

References: vLLM Quantized KV Cache; vLLM April 2026 FP8 KV accuracy and performance research.