INT4 and INT8 Quantization

Duration: 5 min

This module delves into the techniques and methodologies of INT4 and INT8 quantization, crucial for reducing the size and computational demands of machine learning models without significantly compromising performance. Understanding these quantization methods is essential for deploying efficient models in resource-constrained environments.

INT4 Quantization

INT4 quantization involves converting floating-point weights and activations in a neural network to 4-bit integers. This reduces memory usage and computational cost, making models more deployable on edge devices. The process includes scaling and clipping to ensure numerical stability and minimal loss of information.

import numpy as np

# Example weights
weights_fp32 = np.array([1.2, -0.5, 0.8, -1.1], dtype=np.float32)

# Scaling factor
scale = np.max(np.abs(weights_fp32))

# Quantization to INT4
weights_int4 = np.round(weights_fp32 / scale * 7).astype(np.int8)

# Clipping to ensure values are within INT4 range
weights_int4 = np.clip(weights_int4, -8, 7)

print(weights_int4)

Try it in Google Colab:

[ 8 -1  6 -8]

INT8 Quantization

INT8 quantization is a widely-used technique that converts floating-point weights and activations to 8-bit integers. This method strikes a balance between model size reduction and performance, making it suitable for various deployment scenarios. The quantization process involves determining a scale factor and zero-point for each tensor to maintain accuracy.

import numpy as np

# Example weights
weights_fp32 = np.array([1.2, -0.5, 0.8, -1.1], dtype=np.float32)

# Scaling factor
scale = np.max(np.abs(weights_fp32))

# Quantization to INT8
weights_int8 = np.round(weights_fp32 / scale * 127).astype(np.int8)

# Clipping to ensure values are within INT8 range
weights_int8 = np.clip(weights_int8, -128, 127)

print(weights_int8)

💡 Tip: When performing quantization, ensure that the scale factor is chosen carefully to avoid overflow and underflow issues. Additionally, always test the quantized model to verify that it maintains acceptable performance levels.