Framework & Tool
TensorRT-LLM
NVIDIA's LLM-specific extension of TensorRT for maximum GPU inference throughput.
Definition
TensorRT-LLM compiles transformer models to highly-optimised CUDA kernels with INT8/FP8 quantisation, fused operators and in-flight batching. It is the fastest serving path on NVIDIA GPUs and underpins Triton-based production deployments.
Common use cases
- NVIDIA inference
- Triton serving
- High-throughput production