What is quantization-aware training (QAT)?

Quantization-aware training (QAT) teaches a neural network to tolerate the rounding and clipping errors it will face after low-precision deployment. During training, it inserts fake-quantization operations around selected weights and/or activations: values are rounded to a simulated integer grid and immediately dequantized back to floating point, so the network learns under the anticipated low-bit distortions while its trainable parameters can still be updated in floating point.

After QAT, the trained model must be converted and lowered for a compatible quantized inference backend. Running fake quantization during training does *not* automatically mean the training graph runs INT8 arithmetic or consumes only one byte per weight. QAT is a model-accuracy adaptation technique for low-precision inference, not a synonym for quantized training kernels or for mixed-precision FP16/BF16 training.

1. Why do we need QAT?

• Floating-point weights and activations are continuous approximations: An ordinary FP32 model can learn values such as 0.267 or 1.013.
• INT8 has a finite grid: Integer encodings have only 256 possible bit patterns. Mapping float values to that grid introduces rounding, and values outside the calibrated range are clipped.
• Plain post-training quantization (PTQ) quantizes an already-trained floating-point model without additional training to compensate. It is often fast and sufficient, especially when calibration is representative and the model is tolerant to noise.
• Quantization-aware training (QAT) exposes the model to simulated quantization while optimizing weights, allowing it to adapt to these errors. QAT can recover accuracy that PTQ loses, but improvement is not guaranteed and QAT costs training time, data and engineering effort.
• Deployment objective: On suitable hardware, real INT8 kernels may reduce model storage, memory bandwidth, inference latency or energy use. Training with fake INT8 operations alone provides no guaranteed speedup.

The classic study by Jacob et al., *Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inference* (CVPR 2018), demonstrated co-designing quantization and training for efficient on-device integer inference.

2. The quantize → clip → dequantize equations

For a scalar real value x, choose:

• Scale `s > 0`: The real-world interval represented by one integer step.
• Zero point `z`: The integer code for real value 0, usually an integer in the quantized range.
• `q_min, q_max`: The allowed integer boundaries. For a signed INT8 example, use q_min = −128 and q_max = 127.
• Rounding: Map the scaled value to the closest integer under the specified rounding rule.
• Clipping: Saturate codes outside the supported interval.

Quantization:

q = clip(round(x / s) + z, q_min, q_max).

Dequantization:

x_fq = s × (q − z).

Here x_fq is a floating-point value that lies on the quantization grid. A QAT fake-quantization operation produces this x_fq numerically during training rather than permanently replacing every trainable FP32 weight with an INT8 parameter.

Quantization error: e = x_fq − x. Within a nonsaturated uniform grid and round-to-nearest, |e| is typically no more than s/2; clipping outliers can cause much larger error.

3. Beginner-friendly numerical example: Calibrate an asymmetric INT8 grid

Suppose a representative observer or calibration batch measured an illustrative activation range:

• Observed minimum: x_min = −2.5.
• Observed maximum: x_max = 2.6.
• Signed INT8 range: q_min = −128, q_max = 127.

The choice is an asymmetric affine scheme: the zero point need not be zero.

Step 1 — Calculate the scale.

s = (x_max − x_min) / (q_max − q_min)

s = (2.6 − (−2.5)) / (127 − (−128)) = 5.1 / 255 = 0.02.

Step 2 — Calculate the integer zero point.

z = clip(round(q_min − x_min / s), q_min, q_max)

z = round(−128 − (−2.5/0.02)) = round(−128 + 125) = −3.

So real zero is represented by integer −3. Note that a zero point of −3 is not an error; it is a consequence of centering an integer grid over an asymmetric observed float interval.

Step 3 — Quantize and dequantize an in-range value x=0.267.

• Scale the value: x/s = 0.267/0.02 = 13.35.
• Shift by zero point: 13.35 + (−3) = 10.35.
• Round to nearest integer: round(10.35) = 10.
• Clip to INT8 range: q = clip(10,−128,127) = 10.
• Dequantize: x_fq = (10−(−3))×0.02 = 13×0.02 = 0.26.
• Error: `e = 0.26−0.267 = −0.007`.

That 0.267 → 0.26 change is exactly the sort of perturbation fake quantization exposes during QAT.

Step 4 — Verify other in-range values.

• `x = −0.317`: x/s+z = −15.85−3 = −18.85; round to q=−19; dequantize (−19+3)×0.02 = −0.32; error `−0.003`.
• `x = 0`: q=−3; dequantize (−3+3)×0.02=0; error 0.
• `x = 1.013`: x/s+z = 50.65−3 = 47.65; round to q=48; dequantize (48+3)×0.02 = 1.02; error `+0.007`.
• `x = −2.5`: q=−128; reconstructed value −2.5.
• `x = 2.6`: q=127; reconstructed value 2.6.

Step 5 — Observe clipping beyond calibration limits.

• `x=3.1`: Raw integer code 3.1/0.02−3=152; clip to 127; reconstruct 2.6; error `−0.5`.
• `x=−3.0`: Raw code −3/0.02−3=−153; clip to −128; reconstruct −2.5; error `+0.5`.

Critical observation: In-range rounding errors were only a few thousandths, but unrepresented outliers incurred errors of half a real unit. Selecting representative activation ranges and deciding how to treat outliers can matter as much as the number of bits.

These examples avoid exact half-step rounding ties. At a tie, the framework's rounding rule (e.g., ties-to-even) can matter.

4. What are observers and calibration?

An observer collects information about real activation or weight distributions so the quantizer can choose scale and zero point.

• Min/max observer: Tracks observed extrema and derives parameters from the recorded range.
• Moving-average observer: Updates running activation range statistics more gradually, reducing sensitivity to one training minibatch but potentially missing rare extremes.
• Histogram/percentile approach: Can trade a small number of clipped outliers for finer grid resolution over common values, subject to backend/framework support.
• PTQ calibration: A representative input dataset passes through an already-trained model to estimate ranges before conversion; generally no further gradient training compensates for quantization.
• QAT observers: Range statistics can be collected during a warm-up phase or updated as training proceeds. Some workflows freeze observer updates later to stabilize the quantization grid while weights adapt.
• Freeze versus disable fake quantization: Disabling observers stops statistics updates; disabling fake quantization removes simulated rounding. These are different toggles.

Calibration failure mode: Training batches only cover [−2.5,2.6], but production data often contain values around 3.1. The example shows why the deployment quantizer would clip those values. Calibrate on representative data, inspect per-layer clipping and consider outlier-aware alternatives rather than relying on one accidental batch.

5. How can the model train through nondifferentiable rounding?

The rounding function has a derivative of zero almost everywhere, so directly backpropagating the exact derivative of hard rounding would not give a useful learning signal.

A common QAT approximation is the straight-through estimator (STE):

• Forward: Apply rounding and clipping to simulate the target quantizer.
• Backward: Approximate the derivative of fake quantization with respect to the input as 1 within a useful nonsaturated range, passing the gradient through, and commonly 0 outside a clipping range, or use a tailored surrogate rule.
• Floating-point trainable weights: The optimizer updates underlying high-precision weights. Later forward passes reapply the fake quantizer to those modified values.
• Why approximate? STE is a biased surrogate gradient, not the mathematical derivative of rounding. Despite that mismatch, it often makes QAT optimization practical.
• Boundary detail: Framework behavior can differ at quantization/clipping boundaries and with learned scale parameters; do not assume STE is always an unrestricted identity.

Tiny one-weight learning example

For illustration only, reuse the affine grid s=0.02, z=−3 for one scalar trainable value. Actual networks may use different scales for weights and activations.

• Current float parameter: w=0.267. Its fake-quantized forward value is 0.26.
• Suppose the input is a=1, desired output is y*=0.28, and output is y=a×fake_quant(w)=0.26.
• Squared loss: L=(0.26−0.28)²=0.0004.
• With an STE derivative of 1 in this unclipped region: ∂L/∂w ≈ 2(0.26−0.28)×1 = −0.04.
• One illustrative SGD update with learning rate 0.1: w_new=0.267−0.1×(−0.04)=0.271.
• Next fake-quantized weight: 0.271/0.02−3=10.55 rounds to integer 11, dequantizing to (11+3)×0.02=0.28.
• New toy loss: (0.28−0.28)²=0.

Meaning: The *stored floating-point weight* can move across an integer-grid decision boundary, improving its quantized forward behavior even though rounding itself is not differentiable. This is a deliberately constructed single-step example—not a guarantee that QAT always reaches zero loss.

6. Symmetric versus asymmetric quantization

Symmetric quantization

• Quantization levels are centered on zero, usually with `z=0`.
• Often favored for weights because integer arithmetic and zero handling can be simpler.
• For example, with signed symmetric integer codes −127…127 and floating magnitudes up to 2.6, choose s=2.6/127≈0.02047244. The real interval is approximately [−2.6,2.6].
• A small value 0.02 maps to integer 1, reconstructing to about 0.02047244 (error about +0.00047244).

Asymmetric quantization

• The zero point can be nonzero, such as −3 in our main example.
• Useful for activation distributions that are shifted or not symmetric around zero; it may use available codes more efficiently.
• Requires compatible handling of zero-point offsets and integer arithmetic, which can add complexity depending on the backend.

Neither scheme is universally superior. Match quantization parameters to the distribution and the execution kernel's supported integer scheme.

7. Per-tensor versus per-channel quantization

Per-tensor: One shared scale and zero point for the entire weight tensor or activation tensor.

• Simple and compact metadata, often cheaper to implement.
• A large outlier in one output channel can force a coarse scale on small weights in another.

Per-channel: Different scales, and sometimes zero points, for different channels (often the output-channel axis of a convolution or linear weight matrix).

• More accurate when channel ranges vary widely.
• Requires more scale metadata and backend/kernel support.

Numerical example

Suppose one weight channel has values only within ±0.05, while another reaches ±3.0.

• A shared symmetric per-tensor signed-INT8 scale is 3.0/127≈0.02362205.
• The small-channel weight 0.02 then maps to integer 1, reconstructing as 0.02362205, error about +0.00362205.
• A channel-specific scale for the narrow channel is 0.05/127≈0.00039370.
• The same weight maps to integer 51, reconstructing as 0.02007874, error about +0.00007874.
• In this simple case, the per-channel grid substantially reduces this particular weight's quantization error.

This is a numeric demonstration for weights; activation quantization may instead use per-tensor, per-token, per-group or dynamic parameters depending on supported hardware.

8. End-to-end QAT workflow: Prepare, train, convert, deploy

Phase A — Establish a floating-point baseline

1. Train or fine-tune a suitable floating-point model and record accuracy, task metrics, inference latency and size.
2. Determine the *deployment target*: CPU, mobile NPU, accelerator, inference framework, supported kernel operators, and required INT8 layout.
3. Optionally run PTQ first; if it meets the accuracy and performance target, additional QAT may be unnecessary.

Phase B — Prepare for QAT

1. Configure target weight and activation formats, per-tensor/per-channel granularity, observers, clipping ranges and operator fusion.
2. Insert fake quantization around supported operators, or transform the model using the framework's supported QAT preparation workflow.
3. Use representative calibration/training data to collect range statistics; consider freezing observers once scales stabilize.

Phase C — Train with simulated quantization

1. Run ordinary losses and optimizer steps against the fake-quantized forward path, using framework-provided STE/surrogate backward.
2. Keep trainable parameters and necessary optimizer states in a suitable floating-point format; monitor loss spikes, activation saturation, outlier clipping and validation accuracy.
3. Avoid changing ranges unpredictably late in training, and measure whether QAT meaningfully recovers PTQ-induced quality loss.

Phase D — Convert, lower, and validate

1. Convert fake quant modules into actual deployable quantization representations for the intended backend.
2. Lower/compile/export using that backend's compatible INT8 kernels and supported operator set.
3. Compare the actual converted model with float, PTQ and fake-quantized evaluations using the same held-out data.
4. Benchmark actual size, throughput, latency, memory and energy on the target hardware; quantized storage alone does not prove lower runtime latency.

Training-versus-inference distinction: Fake quantization during training keeps numerical values in floating point while simulating a quantization grid. Converted inference may use genuine integer representations, integer accumulators and explicit scale/zero-point handling; operator fusions, rescaling and backend details determine the actual execution.

9. Practical PyTorch example: The exact fake-INT8 arithmetic

PyTorch provides torch.fake_quantize_per_tensor_affine to demonstrate a particular affine quantizer. The following CPU example illustrates fake quantization only, not an entire training or deployment pipeline.

PYTHON
import torch

x = torch.tensor([-3.0, -2.5, -0.317, 0.0,
                   0.267, 1.013, 2.6, 3.1], dtype=torch.float32)

y_fake = torch.fake_quantize_per_tensor_affine(
    x, scale=0.02, zero_point=-3,
    quant_min=-128, quant_max=127
)
print(y_fake)
# approximately:
# tensor([-2.50, -2.50, -0.32, 0.00,
#          0.26, 1.02, 2.60, 2.60])

This function returns a floating-point tensor on the simulated quantization grid. Notice that the outlier 3.1 is clipped to 2.6 and −3.0 is clipped to −2.5.

PyTorch deployment workflow (torchao PT2E / backend example):

PyTorch's current documentation directs new quantization development toward torchao, rather than treating the older torch.ao.quantization flow as the primary future-facing API.

A backend-targeted INT8 QAT route documented by torchao is:

PYTHON
import torch
from torchao.quantization.pt2e.quantize_pt2e import (
    prepare_qat_pt2e, convert_pt2e
)
from executorch.backends.xnnpack.quantizer.xnnpack_quantizer import (
    XNNPACKQuantizer, get_symmetric_quantization_config
)

# 'float_model', 'example_inputs', 'train_loop' are supplied
# by your application; the XNNPACK/ExecuTorch packages
# and compatible versions must be installed.
exported = torch.export.export(float_model, example_inputs).module()
quantizer = XNNPACKQuantizer().set_global(
    get_symmetric_quantization_config(is_qat=True)
)
prepared = prepare_qat_pt2e(exported, quantizer)
train_loop(prepared)                 # real optimization using fake quant
converted = convert_pt2e(prepared)  # quantization representation

# An appropriate export/runtime-specific lowering step remains
# necessary to execute the target integer kernels.

Practical compatibility caution: This is a documented flow skeleton, not a claim that every model, quantization scheme, installed torchao version or device can use the XNNPACK path. The quantizer and native backend must support the model operators and the chosen bit widths. For exported graphs, use the documented exported-model train/eval mode utilities where needed; ordinary .train()/.eval() calls after export may not have the intended effect. The current torchao `quantize_`/QATConfig route is another documented QAT interface; its supported base configs and explicit custom configurations are version-dependent. Do not blindly substitute a weight-only INT4 example for an INT8 deployment goal.

10. How does QAT compare with PTQ?

• PTQ: Quantizes an existing floating-point checkpoint, usually with calibration and no gradient fine-tuning. It is cheaper to adopt and can be highly effective when the model is quantization-tolerant.
• QAT: Introduces simulated quantization into training or fine-tuning, potentially improving quantized accuracy at additional computational and data cost.
• **Accuracy comparison must use the *converted* model:** A fake-quantized training loss improvement does not automatically transfer to the production backend.
• QAT is not universally better: On easy models/PTQ settings the benefit may be marginal; noisy/limited fine-tuning data or poor hyperparameters can degrade accuracy.
• Trainable model and deployment differences: QAT does not guarantee an exact match if the backend applies different rounding, fusion, signedness, accumulator widths or unsupported operators.

11. Hardware trade-offs, efficiency and applications

• Storage: An eligible tensor with 1 million FP32 weights uses 4 MB (decimal); packed INT8 codes use 1 MB of raw weight bytes. That is 4× less for those stored values, not automatically 4× less total model file size or runtime memory.
• Latency: ARM/mobile CPUs, edge NPUs and some GPU/CPU inference runtimes can accelerate compatible INT8 operators. Unsupported layers or quantize/dequantize boundaries can negate gains.
• Accumulation precision: A target INT8 matrix multiplication may accumulate into larger integer types, then requantize. "INT8 inference" does not mean every intermediate arithmetic result remains eight bits.
• Energy and bandwidth: Smaller weights and activations can reduce memory traffic; actual improvements depend on data movement and hardware.
• Applications: On-device vision, classification/detection, mobile audio, compact language-model inference and deployed systems with strict latency/memory constraints.
• Deployment contract: Freeze or explicitly manage observers, verify scales and zero points, run conversion, test the actual exported model, and benchmark target hardware.

12. Common mistakes and limitations

• Thinking fake quantization already creates INT8 tensors: The QAT forward computation commonly uses floating-point tensors with quantized *numerical values*; conversion and backend lowering remain separate.
• Forgetting the zero point: In affine quantization, a real value of zero maps to integer z, which may be nonzero.
• Rounding without clipping: INT8 cannot represent codes outside its allowed bounds; omitting clipping hides serious outlier errors.
• Calibration on unrepresentative data: Production activation distributions can be wider than the observer range, causing saturation.
• Incorrect or frozen-too-early observers: Unstable/misaligned scales may hurt gradients; leaving constantly moving ranges near the end may also impair convergence.
• Inconsistent QAT versus backend formats: A model trained with one signedness, channel axis, zero-point domain or rounding rule may behave differently when lowered to hardware.
• No STE or incorrect gradients through rounding: Naive rounding is nondifferentiable; relying on its exact gradient prevents learning in the expected way.
• STE outside the valid range: Treating heavily clipped outliers like ordinary in-range values can make training misleading; inspect the surrogate gradient and range.
• Assuming every layer should be quantized: Softmax, normalization, small sensitive heads or other operators may need floating-point execution depending on the backend.
• Ignoring per-channel variation: One broad per-tensor scale can erase small weights in narrow channels.
• Comparing only training accuracy: Validate converted-inference accuracy, tail cases, quantization saturation, latency, throughput, size and power.
• Assuming guaranteed savings: INT8-friendly storage is not a promise of 4× faster computation or a corresponding total-memory reduction.
• Confusing training-efficiency topics: QAT is about accuracy under future integer inference, not activation-memory recomputation, gradient accumulation, or structurally removing model weights.

Curriculum boundary: Question 296 explained activation checkpointing and its recomputation–memory trade-off. Question 297 explains fake quantization, affine INT8 arithmetic, observers, STE and deployment-aware adaptation. Question 298 will compare structured versus unstructured neural-network pruning; QAT does not itself remove network channels, weights or blocks.

Key takeaway: QAT trains against simulated quantization errors so a model can learn to survive low-precision deployment. Correct scale, zero point, rounding, clipping, calibration, surrogate gradients and backend conversion matter as much as the number of bits. Use PTQ when it already meets requirements; apply QAT when the measured quantized accuracy justifies additional training.

Research and official documentation: Jacob et al., *Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inference* (CVPR 2018); PyTorch torchao, *Quantization-Aware Training workflow*; PyTorch torchao, *PT2E QAT*; PyTorch, *fake_quantize_per_tensor_affine*; PyTorch, *MinMaxObserver*; PyTorch, *Quantization API migration to torchao*.