How does quantization affect LLM quality?

Quantization can affect LLM quality because reducing numerical precision changes how accurately the model's weights—and sometimes activations or KV-cache values—are represented. These small numerical differences can alter token probabilities and occasionally change the model's final answer.

However, quantization does not automatically make an LLM noticeably worse. Quality loss depends on the model, method, bit width, and task.

Why can quality change?

• Rounding and reconstruction error: A weight such as 0.73 may be approximated as 0.70. Across many layers, such changes can affect intermediate calculations.
• Sensitive weights and outliers: Some channels or layers are more important to model behavior than others. Treating every value with the same precision can hurt accuracy.
• Low bit widths: Fewer representable values usually create more approximation error. A well-quantized 4-bit model can remain useful, but aggressive 2-bit quantization is typically more challenging.
• Quantization design: Weight-only quantization, weight-and-activation quantization, and KV-cache quantization alter different parts of inference and can have different quality effects.

Is 8-bit always better than 4-bit?

Not necessarily on every benchmark. With the same model and a comparable method, 8-bit usually has more numerical precision and often stays close to the FP16/BF16 baseline. Carefully calibrated 4-bit methods, such as GPTQ or AWQ, can also preserve strong quality while using less memory.

The method matters as much as the nominal bit width. One poorly configured 8-bit approach can perform worse on a specific task than a well-tuned 4-bit approach.

A simple example

Consider a multiple-choice question where the unquantized model narrowly prefers answer A over B. Quantization slightly changes the model's scores:

Before quantization: A = 0.51, B = 0.49
After quantization:  A = 0.48, B = 0.52

The preferred answer flips. This is an illustration, not a universal prediction of quantization behavior; many prompts remain unchanged.

How can quality loss be reduced?

• Use suitable scales and group sizes: Per-channel or group-wise quantization can better fit weight distributions than a single global scale.
• Protect sensitive values: Mixed-precision schemes can retain higher precision for outliers or important layers.
• Use calibration data: Methods such as GPTQ and AWQ can account for representative activations when determining how to quantize weights.
• Avoid unnecessary precision reduction: Choose a bit width that meets actual memory and quality requirements.

How should you evaluate a quantized LLM?

1. Keep the baseline fixed: Compare the same model before and after quantization, using the same prompts and decoding settings.
2. Check task quality: Evaluate factual Q&A, reasoning, coding, structured extraction, or domain-specific tasks relevant to your application.
3. Measure next-token behavior: Perplexity can help identify language-modeling changes, but it cannot substitute for application-level evaluation.
4. Inspect failures: Review incorrect answers, formatting errors, and difficult or long-context inputs instead of relying only on one average score.
5. Measure the benefit: Record peak GPU memory, latency, and throughput alongside quality metrics.

A model with nearly unchanged benchmark accuracy may still fail on a small but important subset of production examples. A quality-versus-memory decision is workload-specific.

See “What is LLM quantization, and how does it work?”, “What is 8-bit LLM quantization?”, and “What is 4-bit LLM quantization?” for the underlying techniques.

In short: quantization introduces numerical approximation that *may* reduce LLM quality, especially at very low precision. Good quantization methods can retain much of the original performance, but production suitability must be established by evaluation.

References: Hugging Face: Selecting a quantization method, GPTQ, and AWQ.