INT8 Quantization

Understanding how to do calculations in INT8 instead of FP32 and still get same results is a core optimization in DL. DL is highly insensitive to local errors.

INT8 Quantization is a technique to run DL models by computing arithmetic using INT8 inputs (instead of FP32) with INT8 outputs. This reduces the memory bandwidth by 4X and improves computation by up to 3X (using AVX512 VNNI instructions).

For AVX details, see chapter on “Matrix Multiplication” and “Convolution”.

By default, weights are in FP32 and computation is in FP32 as well. These are converted to INT8. The new pre-training model is a quantized INT8 model.

There are 3 basic approaches to Quantization (preparing INT8 model):

Core concepts:

• FP32 input mapped to INT8 range
• Core operations: quantize, dequantize, clip
• Symmetric Quantization
• Asymmetric Quantization

FP32 input is mapped to INT8 range

Note: The distribution of numbers remains the same. Numbers can fall out of the INT8 range after conversion is clipped to the max or min INT8 value.

This involves 2 core metrics:

• Scale factor S
• Zero-point Z

Quant/ Dequant operations

2 core operations using S and Z (X is input):

• Quantizeoperation: Convert FP32/INT32 inputs to INT8: X_q = round(X/S) + Z
  • Output is clipped to fit in INT8 range [-128, 127] or [0, 255]
• Dequantizeoperation: Convert INT8 to FP32/INT32: X_d = (X_q - Z) * S
  • Output is clipped to the actual min / max of the input
  • Actual min / max is maintained through the model.

S and Z is computed as:

Symmetric Quantization

If input has range [X_min, X_max] where the middle element is 0, then:

Scale = \(\frac{\mathbf{MAX\ }\left( \left| \mathbf{X}_{\mathbf{\min}} \right|\mathbf{,\ \ \ }\left| \mathbf{X}_{\mathbf{\max}} \right| \right)}{\mathbf{2}^{\mathbf{N - 1}}\mathbf{- 1}}\) or \(\frac{\mathbf{MAX\ }\left( \left| \mathbf{X}_{\mathbf{\min}} \right|\mathbf{,\ \ \ }\left| \mathbf{X}_{\mathbf{\max}} \right| \right)}{\mathbf{q}_{\mathbf{\max}}}\)

Zero point = 0 (always)

Main points:

• INT8 range is [-127, 127] (Input is mapped to this range)
• Use when input data is centered around zero (not + or – skewed)
• Efficient computation as zero-point addition is avoided.
• In relative symmetric input, gives better accuracy as it captures small values better (example, weights in CNN)

Asymmetric Quantization

S = \(\frac{\mathbf{X}_{\mathbf{MAX}}\mathbf{-}\mathbf{X}_{\mathbf{MIN}}}{\mathbf{2}^{\mathbf{N}}\mathbf{- 1}}\mathbf{\ \ or\ }\frac{\mathbf{X}_{\mathbf{MAX}}\mathbf{-}\mathbf{X}_{\mathbf{MIN}}}{\mathbf{q}_{\mathbf{MAX}}\mathbf{-}\mathbf{q}_{\mathbf{MIN}}}\mathbf{\ }\)

Z = ROUND(\(\mathbf{\ }\frac{\mathbf{- \ Xmin}}{\mathbf{S}}\) ) or ROUND( \(\mathbf{q}_{\mathbf{MIN}}\mathbf{-}\frac{\mathbf{Xmin}}{\mathbf{S}}\) )

Main points:

• INT8 range is [-128, 127] (if signed output is needed) or [0, 255]
• Captures full dynamic range of input.
• Better accuracy for highly skewed input (ReLU output)

There are 2 main methods of preparing an INT8 model:

• PTQ
• QAT

Post Training Quantization (PTQ)

Main steps:

• Take a pre-trained FP32 model.
• For weights, calculate the range and compute the scale and zero point accordingly using symmetric quantization.
• For activations, run inference on a subset of the training dataset (called calibration dataset) and measure the range of activations. Clip the outliers using histogram-based calibration and compute the scale and zero point according using asymmetric quantization. This step is called calibration.
• Run inference using the computed scale and zero point.
  • If accuracy is low, update the calibration step.
  • As PTQ is sensitive, first Conv layer must run in FP32 and specific ops like SoftMax should run in FP32. Fuse specific ops like Conv + BatchNorm + ReLU.
• If accuracy remains low, QAT must be used.

Quantization-Aware Training (QAT)

Main steps:

• Take the FP32 model.
• Before each op which needs to be quantized (like Conv), add a Fake Quantization op (that is Quantize -> Dequantize one after another). This introduces INT8 quantization noise in training process.
• In forward pass, ops like Conv run in the default FP32 precision.
• In backward pass, the loss gradient is bypassed through the Fake Quantization. This is called straight-through estimator STE. Weights are updated as usual.
• At the end of the training process, the Fake Quantization nodes hold the scale and zero point.

Calibration

• Process: clip original weights -> then quantization (avoid outlier + increase representation) -> improves accuracy.
• Manually choose percentile of weights to consider.
• Minimize mean squared error (MSE) between original and quantized weight.
• Minimizing entropy (KL-divergence) between original and quantized value.

3 variants:

• Smooth Quant
• Weight Only
• Activation Aware Quantization

Smooth Quant

Rescale weights and activations to smooth their dynamic ranges, preventing large values in activations from dominating and causing clipping.

• Improves INT8 quantization accuracy especially in transformer models.
• Avoids outlier activation issues without retraining.
• Compatible with post-training quantization (PTQ).

Weight Only

Only weight is quantized to FP8/INT8.

• Reduces model size.
• No need for activation calibration or retraining.
• Allows high-speed inference on GPUs with fast FP16 compute units.
• Doesn't reduce activation memory footprint (which matters for long-context LLMs).
• May degrade accuracy if quantization noise in weights is too high.

Activation-Aware Quantization (AAQ)

Takes into account the distribution of activations during calibration or training to minimize quantization error.

• Helps preserve model accuracy.
• Often paired with per-channel scaling and min-max clipping.
• Used in Quantization-Aware Training (QAT) or calibration-based PTQ.