Technique
KV Cache Quantization
Quantising attention keys/values to INT8 or INT4 to shrink KV-cache memory.
Definition
Since the KV cache dominates LLM inference memory, quantising it to 8-bit or 4-bit per element lets servers fit much longer contexts or more concurrent users. Modern engines (vLLM, TRT-LLM) support FP8 and INT4 KV cache with negligible quality loss.
Common use cases
- Long-context serving
- Multi-tenant inference
- Memory savings