Infrastructure
FP8
8-bit floating-point format used on H100/B200 to double FP16 throughput at minor quality cost.
Definition
FP8 (E4M3 and E5M2 variants) halves the bit-width of FP16 while preserving an exponent, allowing larger dynamic range than INT8. NVIDIA's Transformer Engine auto-casts to FP8 in training and inference. Frontier 2024–2025 training increasingly runs in FP8.
Common use cases
- Frontier training
- Fast inference
- TensorRT-LLM