Quantization

Executive Definition: Quantization — The process of mapping continuous infinite values to a smaller set of discrete finite values, used in AI to reduce the precision of model weights to decrease memory usage and improve inference speed.

In the context of deep learning, quantization involves converting neural network parameters from high-precision floating-point numbers (such as FP32 or FP16) to lower-precision integers (such as INT8 or INT4). This compression dramatically shrinks the physical file size of the model and reduces the memory bandwidth required during execution, enabling large language models to run on consumer hardware.

Frequently Asked Questions

Does quantization hurt model quality?

Reducing numerical precision inherently causes some information loss. However, techniques like 8-bit or 4-bit quantization often result in negligible degradation on practical benchmarks.

What are some common quantization formats?

Popular formats include GGUF for local CPU inference, and AWQ or GPTQ which are optimized for high-throughput GPU inference.

Can any model be quantized?

While most standard transformer architectures support quantization, highly sensitive models or those requiring strict mathematical precision may suffer notable degradation. Pre-quantized models are widely available.