Concept
Inference
Running a trained model on new inputs to produce predictions or generations.
Definition
Inference is the production phase of a model's lifecycle, after training. Its cost is dominated by GPU memory bandwidth and arithmetic throughput; optimisations include quantisation, KV-cache reuse, speculative decoding, and batching. Inference latency and token throughput are the headline metrics for serving systems.
Common use cases
- Production serving
- Batch jobs
- Edge inference