Infrastructure
Model Parallelism
Splitting a model's layers or tensors across multiple GPUs that do not fit one.
Definition
Model parallelism partitions a model whose memory exceeds one GPU. Variants split by layers (pipeline parallelism) or by tensor dimensions (tensor parallelism); modern training combines them. It is essential beyond ~10B parameters.
Common use cases
- Large models
- Frontier training
- MoE routing