Infrastructure
INT4
4-bit integer weight quantisation used widely for fast local LLM inference.
Definition
INT4 (and the NF4 normal-float variant) shrinks weight memory by 4Ă versus FP16 with small quality loss. GPTQ, AWQ and bitsandbytes implement it. INT4 is the default precision for llama.cpp, Ollama and most consumer-GPU and edge deployments.
Common use cases
- Local inference
- Edge devices
- Consumer-GPU serving