Technique
Quantization
Compressing model weights to lower-precision integers (INT8/INT4) to save memory and speed inference.
Definition
Quantisation maps 16-bit weights to 8-, 4- or even 2-bit integers with minimal quality loss. Post-training quantisation is the cheap route; QAT (quantisation-aware training) preserves more accuracy. GGUF, AWQ, GPTQ and INT4 are common runtime formats.
Common use cases
- Edge deployment
- Self-hosting
- Memory reduction