Metric & Benchmark
Throughput
Tokens or requests processed per second by an inference system.
Definition
Throughput measures system-level capacity — total useful work per unit time. For LLMs it is typically reported in tokens/second across the whole batch, distinct from per-user latency. Maximising throughput often trades off against per-request latency.
Common use cases
- Capacity planning
- Cost modelling
- Serving design