Framework & Tool
vLLM
High-throughput LLM serving engine with paged-attention KV-cache management.
Definition
vLLM is an open-source inference engine pioneered by Berkeley that introduced PagedAttention â virtual-memory-style KV cache â yielding 2â5Ă higher throughput than naive serving. It supports streaming, batching, quantisation and most open-weights models.
Common use cases
- Production LLM serving
- Self-hosting
- Inference benchmarks