Infrastructure
Gradient Accumulation
Summing gradients over multiple micro-batches before applying an optimiser step.
Definition
Gradient accumulation simulates a large effective batch size on memory-constrained hardware: run k micro-batches, accumulate gradients, then step. It is essential for training big models on a single GPU and for fine-tuning under QLoRA setups.
Common use cases
- Single-GPU fine-tuning
- Large effective batches
- Memory-constrained training