What is 8-bit LLM quantization?

8-bit LLM quantization stores supported model weights using 8-bit values—typically INT8—instead of 16-bit floating-point values. This can roughly halve raw weight memory compared with FP16 or BF16, helping large language models fit onto smaller GPUs.

Unlike general quantization, this question focuses on how 8-bit inference works and why implementations sometimes keep sensitive calculations at higher precision.

How does INT8 quantization work?

A floating-point weight is mapped to a nearby 8-bit integer using a scale factor:

original weight: 0.73
scale:           0.10
stored INT8:     round(0.73 / 0.10) = 7
approximation:   7 × 0.10 = 0.70

Because 0.70 differs from 0.73, quantization introduces rounding error. Practical implementations use scales for groups or vectors of weights to reduce that error.

What is special about LLM.int8()?

The LLM.int8() method uses mixed-precision matrix multiplication:

• Most values are processed using 8-bit operations.
• Sensitive outlier features are handled separately in 16-bit precision.
• This helps preserve output quality, rather than forcing every computation into INT8.

Other 8-bit quantization methods may use different calibration, scaling, or computational approaches. 8-bit weight storage does not imply that every activation or operation is INT8.

How much memory can it save?

For an illustrative 7-billion-parameter model:

FP16 weights: 7 billion × 2 bytes ≈ 14 GB
INT8 weights: 7 billion × 1 byte  ≈  7 GB

These are raw weight-storage estimates. Real GPU usage is higher because some layers remain unquantized and the runtime also needs scales, activations, workspace memory, and KV cache.

Practical example using Transformers

PYTHON
from transformers import AutoModelForCausalLM, BitsAndBytesConfig

quant_config = BitsAndBytesConfig(load_in_8bit=True)

model = AutoModelForCausalLM.from_pretrained(
    "facebook/opt-350m",
    quantization_config=quant_config,
    device_map="auto",
)

This loads supported layers with bitsandbytes 8-bit quantization when the installed software and hardware support it. It is an inference example, not an instruction to retrain the model.

Does INT8 always improve speed?

No. Smaller weights reduce memory traffic, but measured latency depends on GPU support, optimized kernels, batch size, the extra work for outlier handling, and the decoding workload. Benchmark quality, latency, throughput, and total memory before choosing a production configuration.

8-bit vs. 4-bit: 4-bit weights can use roughly half the raw storage of 8-bit weights, but may introduce greater quantization error and different kernel overhead. Use the least aggressive precision that meets the application's hardware and quality requirements.

See “What is LLM quantization, and how does it work?” for the overview, and “Why does KV cache consume significant memory?” for a memory cost that weight-only quantization does not automatically shrink.

In short: 8-bit quantization cuts model-weight storage by approximately 50% relative to FP16 while aiming to preserve output quality; its real inference benefit depends on the implementation and hardware.

References: Hugging Face bitsandbytes documentation, LLM.int8() paper (arXiv:2208.07339), and MachinoAI LLM.int8() research.