“Quantisation converts model weights and activations from high-precision formats, such as FP32, into lower-precision formats, such as FP16, BF16, INT8, or INT4.
The number of bits determines how many values a format can represent. FP32 offers high precision and is a common training baseline. FP16 and BF16 use half as much memory. BF16 retains the eight-bit exponent of FP32, so it has a similar numerical range, but it provides lower precision because its mantissa is shorter. INT8 provides 256 integer codes, while INT4 provides only 16.
Consider a model with 18 million parameters. With FP32, each parameter requires four bytes, giving a model size of 72 megabytes. FP16 or BF16 reduces this to 36 megabytes. INT8 reduces it to 18 megabytes, and INT4 reduces it to approximately 9 megabytes.
Quantisation offers three main benefits. First, it reduces model size and memory traffic. Second, supported hardware can execute low-precision operations faster. Third, integer operations generally consume less energy than floating-point operations. These advantages make quantisation important for mobile devices, embedded systems, and edge AI.
Uniform quantisation maps floating-point values to a limited set of integer codes. A common equation is:
q=round(x/s)+z
Here, xis the original value, sis the scale, and zis the zero-point. We can approximately reconstruct the value using:
x ̂=s(q-z)
The difference between xand x ̂is the quantisation error. Lower bit widths provide fewer representable levels, so the error normally becomes larger.
Symmetric quantisation fixes the zero-point at zero and uses a range centred around zero. It simplifies computation and works well for weight distributions that are approximately symmetric. Asymmetric quantisation uses a non-zero zero-point, allowing the integer range to shift. This normally fits skewed activation distributions, such as non-negative outputs after ReLU, more effectively.
Quantisation also differs in granularity. Per-tensor quantisation uses one scale and zero-point for the entire tensor. It is simple and hardware efficient, but a few large values can reduce precision for all the smaller values. Per-channel quantisation gives each channel its own quantisation parameters. It usually improves accuracy for CNN weights. Per-group quantisation divides channels into smaller groups. It fits local ranges more accurately and is commonly used for low-bit Transformer and large language model weights, although it requires more metadata.
For activation quantisation, we can use dynamic or static methods. Dynamic quantisation calculates the activation scale during inference from the current input. It adapts well when activation ranges change, but it introduces runtime overhead. Static quantisation estimates the ranges in advance using a representative calibration dataset. This enables faster integer inference, but poor calibration can lead to clipping and larger errors.
Post-training quantisation, or PTQ, quantises an already trained model. It requires little or no additional training, so it is fast and convenient. However, the model has never experienced quantisation error during training. Accuracy may therefore decrease, especially at very low precision.
Quantisation-aware training, or QAT, simulates quantisation during the forward pass. The model keeps floating-point master parameters for optimisation, but fake-quantisation operations introduce rounding and clipping effects during training. Because rounding has zero gradients in most places, QAT commonly uses the straight-through estimator during backpropagation.
At the beginning, PTQ and QAT can produce the same quantised forward result. The difference appears during training. QAT calculates gradients from the quantised simulation, updates the floating-point weights, and quantises them again during the next forward pass. The model gradually adjusts its parameters to values that remain effective after quantisation.
In summary, PTQ is simpler and faster to apply, while QAT usually preserves accuracy more effectively. The final choice should consider the target hardware, required bit width, acceptable accuracy loss, and available training resources.”
