Infrastructure
Data Parallelism
Replicating the model on every GPU and splitting the batch across them.
Definition
In data parallelism each GPU holds a full copy of the model and processes a shard of the batch; gradients are all-reduced before the optimiser step. It is the simplest distributed strategy and the default for models that fit in one GPU's memory.
Common use cases
- Small-to-medium models
- Multi-GPU servers
- FSDP/DDP