Speculative Decoding
LLM token generation is inherently sequential - each token depends on all previous tokens, so generation cannot be fully parallelised. Speculative decoding breaks this bottleneck using a clever trick: a small, fast "draft" model generates a candidate sequence of tokens in parallel, which the large target model then verifies in a single forward pass. The result is 2–4× lower latency with identical output quality.
The Core Insight
The large model's forward pass can verify multiple tokens in parallel (it is the sequential sampling that is slow). If a small draft model can guess the next K tokens correctly most of the time, the large model can verify all K in one pass - equivalent to generating K tokens at the cost of one forward pass.
How It Works
Draft
A small model (e.g. 7B) generates K speculative tokens (typically 4–8) given the current context. This is fast - the small model is cheap.
Verify
The large target model (e.g. 70B) runs a single forward pass over the context plus all K draft tokens simultaneously, producing K+1 probability distributions.
Accept or reject
Each draft token is accepted if the target model's distribution agrees (within a threshold) and rejected otherwise. Accepted tokens are emitted instantly; the first rejected token is resampled from the target model.
Repeat
Process repeats from the accepted position. On average, more than half the draft tokens are accepted - giving 2–4× speedup over naive decoding.
Output Quality Guarantee
Speculative decoding is mathematically guaranteed to produce output with the same distribution as the target model alone. This is not an approximation - if configured correctly, a user cannot tell whether speculative decoding was used. The acceptance sampling step ensures rejected tokens are corrected exactly.
When It Is Most Effective
High acceptance rate (fast)
- • Predictable, formulaic text (code, structured formats)
- • Continuation of long existing passages
- • Low-temperature generation (less diversity)
- • Domain where draft model was trained
Low acceptance rate (less benefit)
- • Creative writing with high temperature
- • Very short responses (not enough tokens to amortise overhead)
- • Domain gap between draft and target model
Variants
- EAGLE (draft model trained on target residuals) - higher acceptance rate than a standalone small model, widely used with Llama 3 family
- Self-speculative decoding - uses early exit from the target model's own layers as the draft, removing the need for a separate model
- Medusa - adds multiple prediction heads to the target model itself, no separate draft model required
Serving System Support
Speculative decoding is available in:
- vLLM: enable with
--speculative-modeland--num-speculative-tokens - TensorRT-LLM: supported natively for major model families
- Ollama: not yet supported as of mid-2026