Technique
Speculative Decoding
Acceleration where a tiny draft model proposes tokens that the main model verifies in parallel.
Definition
Speculative decoding pairs a fast small draft model with the target model. The draft emits several tokens; the target verifies all of them in a single forward pass and rolls back where it disagrees. The result is 2–3× faster inference with identical outputs.
Common use cases
- Latency reduction
- Cost reduction
- vLLM/TGI optimisation