AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration
AWQ uses activation statistics to identify salient weight channels and scale them before low-bit weight-only quantization, preserving accuracy while enabling hardware-efficient LLM inference.
Key takeaway
AWQ protects the small fraction of weight channels that matter most by using activation-aware scaling rather than mixed precision or backpropagation-based reconstruction.
Updated on Oct 8, 2026
01
Abstract
- AWQ is a hardware-friendly low-bit weight-only quantization method for LLMs.
- Only a small fraction of weights are especially sensitive to quantization.
- Activation magnitude is used to identify salient input channels.
- Scaling salient channels replaces irregular FP16 protection with a hardware-friendly transformation.
- AWQ needs no backpropagation or reconstruction optimization.
- The paper combines AWQ with TinyChat for efficient edge inference.
02
Introduction
- LLM parameter memory creates a major serving barrier on edge devices.
- Autoregressive generation is often memory-bandwidth bound.
- W4A16 reduces weight traffic while keeping activations in FP16.
- On-device inference can reduce latency, cloud cost, and data exposure.
- AWQ targets generalization rather than fitting one calibration distribution.
- TinyChat turns compression into practical throughput improvements.
03
Problem
- RTN treats all weights as equally important.
- Weight magnitude alone is not a reliable importance signal.
- Large activation channels amplify the effect of their associated weights.
- Keeping salient weights in FP16 improves accuracy but creates mixed-precision hardware overhead.
- Reconstruction-based PTQ can overfit calibration data.
- AWQ must protect salient channels while retaining regular INT3/INT4 execution.
04
Background
- PTQ compresses a trained model without retraining it.
- RTN is the simple round-to-nearest baseline.
- GPTQ uses approximate second-order information and reconstruction.
- AWQ is weight-only, with FP16 activations during inference.
- Group-wise quantization shares scales across fixed-size weight groups.
- W4A16 and W3A16 are central deployment configurations.
05
Methodology
- Calibration collects activation statistics for each relevant input channel.
- High activation magnitude identifies salient channels.
- The paper shows that protecting roughly 1% salient weights can sharply reduce error.
- AWQ scales salient weight channels upward and inversely scales the corresponding input channels.
- The scale preserves the original full-precision matrix product before quantization.
- A grid search selects scaling strength without gradient-based optimization.
Protecting salient weights through activation-aware scaling
Panel (a) shows ordinary round-to-nearest quantization, with perplexity 43.2 in this OPT-6.7B INT3-g128 example. Panel (b) uses activation statistics to identify salient weights: keeping roughly 1% in FP16 lowers perplexity to 13.0, but creates a mixed-precision representation that is less convenient for efficient hardware execution. Panel (c) illustrates AWQ’s alternative: scale salient channels before quantization, with compensating activation scaling, to reduce error while retaining a uniform low-bit weight representation. It achieves the same displayed perplexity of 13.0 in this example. Lower perplexity is better. The key lesson is that a weight’s importance depends on the activations it multiplies, not just its absolute magnitude. This figure uses 3-bit grouped quantization; it is a separate experiment from the 4-bit deployment overview.
Source: AWQ paper, Figure 2; user-supplied excerpt.
06
Architecture
- AWQ consists of calibration, scale search, weight transformation, and low-bit quantization.
- The transformed weights use grouped INT3 or INT4 representation.
- Inference keeps activations in FP16 while loading quantized weights.
- TinyChat adds fused kernels and platform-aware weight packing.
- The same quantized representation can target desktop and mobile hardware.
- Algorithmic search and runtime execution are deliberately separated.
From AWQ compression to TinyChat deployment
The center illustrates a language model moving from FP16 weights to a compact INT4 representation through AWQ. Beneath it, the diagram separates the quantization algorithm, AWQ, from the inference system, TinyChat, which executes the compressed model. The surrounding devices illustrate deployment targets including a Jetson Orin Nano, an ARM CPU Raspberry Pi, an Apple M1 MacBook, and CPU/GPU PCs. The supplied caption reports a 3–4× performance improvement over FP16 in the authors’ evaluated setting and describes a Jetson-based TinyChat computer with 8 GB of memory and 15 W power consumption. These are reported deployment results and device specifications, rather than a guaranteed speedup on every platform. Weight compression reduces storage, while practical acceleration also depends on the inference implementation and hardware.
Source: AWQ paper, Figure 1; user-supplied excerpt.
07
Dataset
- Calibration uses a small representative text set rather than training data.
- The calibration-efficiency study uses OPT-6.7B with INT3-g128.
- AWQ reaches good performance with 16 sequences versus about 192 for GPTQ in the reported comparison.
- PubMed Abstracts and Enron Emails from The Pile test calibration-distribution robustness.
- WikiText-2 is used for language-model perplexity evaluation.
- MBPP, GSM8K, COCO captioning, and 11 VLM benchmarks test broader capabilities.
08
Training
- AWQ is post-training and does not update foundation-model parameters.
- No backpropagation is required for scale learning.
- No layer-wise reconstruction loss is optimized.
- Activation statistics are collected offline.
- INT3 and INT4 grouped quantization are the main settings.
- AWQ can be combined with GPTQ for stronger INT2 results.
09
Experiments
- OPT models support controlled perplexity comparisons.
- LLaMA and Llama-2 experiments span 7B to 70B.
- RTN and GPTQ are the main accuracy baselines.
- Vicuna tests instruction-tuned generalization.
- OpenFlamingo and VILA test multimodal generalization.
- TinyChat is evaluated on RTX 4090, Jetson Orin, RTX 4070, and Raspberry Pi 4.
Evidence highlights
- 01OPT models support controlled perplexity comparisons
- 02LLaMA and Llama-2 experiments span 7B to 70B
- 03RTN and GPTQ are the main accuracy baselines
- 04Vicuna tests instruction-tuned generalization
- 05OpenFlamingo and VILA test multimodal generalization
- 06TinyChat is evaluated on RTX 4090Jetson OrinRTX 4070, and Raspberry Pi 4
10
Baselines
- RTN directly rounds weights to the nearest low-bit value.
- GPTQ uses second-order error compensation and reconstruction.
- 1% FP16 salient-weight protection provides an accuracy reference.
- Random and weight-magnitude selection are compared with activation-based selection.
- AutoGPTQ and ExLlama are runtime comparison points on edge hardware.
- Hugging Face FP16 is the primary full-precision runtime baseline.
Evidence highlights
- 01RTN directly rounds weights to the nearest low-bit value
- 02GPTQ uses second-order error compensation and reconstruction
- 031% FP16 salient-weight protection provides an accuracy reference
- 04Random and weight-magnitude selection are compared with activation-based selection
- 05AutoGPTQ and ExLlama are runtime comparison points on edge hardware
- 06Hugging Face FP16 is the primary full-precision runtime baseline
11
Results
- AWQ consistently improves over RTN and GPTQ in reported LLaMA evaluations.
- OpenFlamingo INT4-g128 AWQ reaches 80.53 CIDEr at 32-shot versus 77.13 RTN and 74.98 GPTQ.
- AWQ reduces the INT4 32-shot degradation from 4.57 to 1.17 CIDEr points.
- VILA-7B and VILA-13B remain near FP16 across 11 visual-language benchmarks.
- CodeLlama-7B-Instruct AWQ reaches 40.64 MBPP pass@1 in the reported table.
- TinyChat reports up to 3.9× speedup on RTX 4090 and 3.5× on Jetson Orin.
Evidence highlights
- 01AWQ consistently improves over RTN and GPTQ in reported LLaMA evaluations
- 02OpenFlamingo INT4-g128 AWQ reaches 80.53 CIDEr at 32-shot versus 77.13 RTN and 74.98 GPTQ
- 03AWQ reduces the INT4 32-shot degradation from 4.57 to 1.17 CIDEr points
- 04VILA-7B and VILA-13B remain near FP16 across 11 visual-language benchmarks
- 05CodeLlama-7B-Instruct AWQ reaches 40.64 MBPP pass@1 in the reported table
- 06TinyChat reports up to 3.9× speedup on RTX 4090 and 3.5× on Jetson Orin
12
Ablation
- Activation-based salient selection outperforms raw weight-magnitude selection.
- Random FP16 protection is weaker than activation-selected protection.
- 1% FP16 protection validates that a tiny subset drives substantial error.
- Scaling reproduces much of the protection benefit without mixed-precision storage.
- AWQ needs roughly 10× fewer calibration sequences than GPTQ in the reported study.
- Under distribution shift, AWQ increases perplexity by only about 0.5–0.6 versus GPTQ's 2.3–4.9 increase.
Evidence highlights
- 01Activation-based salient selection outperforms raw weight-magnitude selection
- 02Random FP16 protection is weaker than activation-selected protection
- 031% FP16 protection validates that a tiny subset drives substantial error
- 04Scaling reproduces much of the protection benefit without mixed-precision storage
- 05AWQ needs roughly 10× fewer calibration sequences than GPTQ in the reported study
- 06Under distribution shiftAWQ increases perplexity by only about 0.5–0.6 versus GPTQ's 2.3–4.9 increase
13
Limitations
Error Analysis
- RTN becomes particularly fragile as bit width falls to 3 bits.
- Over-scaling can increase weight dynamic range and hurt quantization.
- Under-scaling fails to protect salient channels sufficiently.
- Reconstruction methods can overfit the calibration distribution.
- INT2 remains substantially harder than INT3/INT4.
- Runtime gains depend on efficient kernels and reducing non-quantized overhead.
14
Conclusion
- AWQ identifies a small set of computation-critical channels using activation statistics.
- Scaling is more hardware-friendly than storing arbitrary FP16 salient weights.
- The method needs no backpropagation or weight reconstruction.
- It generalizes across pretrained, instruction-tuned, coding, math, and multimodal workloads.
- AWQ can complement GPTQ for extreme low-bit quantization.
- TinyChat demonstrates that quantization can produce real edge-device speedups.
15
References
- Original preprint: arXiv:2306.00978.
- Conference publication: Proceedings of Machine Learning and Systems 6, MLSys 2024.
- Official implementation: MIT Han Lab llm-awq repository.
- GPTQ is the main reconstruction-based PTQ comparison.
- LLaMA/Llama-2, Vicuna, CodeLlama, OpenFlamingo, and VILA form the major evaluation models.
- TinyChat is the accompanying efficient inference framework.
Continue reading
Related research
QLoRA
QLoRA combines 4-bit NF4 quantization with LoRA adapters to fine-tune very large language models under a single-GPU memory budget.
LLM QuantizationQuIP
QuIP makes two-bit post-training quantization viable for large language models by combining adaptive rounding with incoherence processing.
LLM QuantizationGPTQ
A one-shot, approximate second-order post-training quantization method that compresses 175B-scale language models to 3–4 bits with minimal accuracy loss.