Infrastructure
Pipeline Parallelism
Splitting a model's layers into stages, with micro-batches pipelined across GPUs.
Definition
Pipeline parallelism assigns successive layers to different GPUs, then streams micro-batches through them to keep all stages busy. Schedules (GPipe, 1F1B, Interleaved) reduce idle bubbles. It is cheaper to interconnect than tensor parallelism but adds latency.
Common use cases
- Frontier training
- Cross-node parallelism
- Mixed parallelism