Infrastructure
Batch Size
Number of training examples processed in parallel in one forward+backward pass.
Definition
Batch size trades off gradient noise against memory and throughput. Larger batches stabilise training but require more memory or gradient accumulation. Inference batching, by contrast, packs many independent requests into one GPU pass to raise throughput.
Common use cases
- Training tuning
- Inference serving
- Cost optimisation