Framework & Tool
vLLM
High-throughput LLM serving engine with paged-attention KV-cache management.
Definition
vLLM is an open-source inference engine pioneered by Berkeley that introduced PagedAttention — virtual-memory-style KV cache — yielding 2–5× higher throughput than naive serving. It supports streaming, batching, quantisation and most open-weights models.
Common use cases
- Production LLM serving
- Self-hosting
- Inference benchmarks