Framework & Tool
aka Text Generation Inference
TGI
HuggingFace's optimised LLM serving stack used by their Inference Endpoints.
Definition
Text Generation Inference is HuggingFace's Rust+Python serving engine, supporting continuous batching, FlashAttention, quantisation, and tensor parallelism. It powers HF's Inference Endpoints and is a common self-host alternative to vLLM.
Common use cases
- Self-hosted inference
- HuggingFace Endpoints