Infrastructure
Tensor Parallelism
Splitting weight matrices across GPUs and synchronising on every layer's forward and backward.
Definition
Tensor parallelism shards individual weight matrices column- or row-wise across GPUs within a node, requiring high-bandwidth interconnects like NVLink. Megatron-LM popularised the technique; it is now standard in vLLM, TGI and TRT-LLM for serving large models.
Common use cases
- Serving large models
- Megatron-style training
- Multi-GPU inference