Infrastructure
KV Cache
Stored attention keys and values from prior tokens, reused so generation does not recompute them.
Definition
During autoregressive generation each new token attends to all previous keys and values. Caching them across steps converts an O(nÂČ) recompute into O(n). The KV cache dominates inference memory; PagedAttention (vLLM) manages it efficiently across many concurrent requests.
Common use cases
- LLM serving
- Memory optimisation
- Throughput tuning