Infrastructure
Distributed Training
Training a model across many GPUs or nodes via data, model or pipeline parallelism.
Definition
Distributed training scales a single training job across hundreds or thousands of accelerators. Strategies — data parallelism, tensor parallelism, pipeline parallelism, expert parallelism — are usually combined (3D / 4D parallelism). Frameworks: DeepSpeed, FSDP, Megatron-LM, JAX/XLA.
Common use cases
- Frontier pre-training
- Large fine-tunes
- Foundation models