Speculative decoding for faster LLM inference
Autoregressive decoding has a serial dependency. The model needs the current token before it can produce the next one.
Speculative decoding does not remove that dependency. It finds cheap guesses for several future tokens, then asks the expensive target model to verify them together.
When enough guesses survive verification, one target-model step advances the sequence by several tokens.
Ordinary decoding spends a pass per token
After prefill, the target model holds a KV cache for the prompt.
For each output token, it reads the latest state, runs the transformer, samples one token, appends new keys and values, and repeats.
The matrix shapes during decode can be small. Model weights and cache data still need to move, so the GPU may have unused arithmetic capacity.
This is why batching helps throughput. Several sequences can share a larger device operation.
A single low-traffic request cannot create that batch by itself. Speculation tries to create useful parallel work within one sequence.
The distinction between prefill and decode matters because speculation targets the decode loop.
A draft proposes and the target verifies
Suppose a draft method proposes four tokens.
The target model evaluates those proposed positions in one pass. The acceptance procedure compares draft and target probabilities and decides how far the sequence can advance.
If the first three tokens are accepted and the fourth is rejected, the target supplies a corrected token and decoding continues.
The target remains the authority. The draft is useful only because a proposal can be cheaper than a target-model step.
The draft may be a smaller model, extra prediction heads, n-gram matching, a suffix cache, or another mechanism.
Exact sampling is the important result
Fast Inference from Transformers via Speculative Decoding describes an acceptance method that preserves the target model's output distribution.
The paper reported a 2 to 3 times acceleration on its T5-XXL experiments with identical output distribution.
That guarantee depends on the algorithm. Greedily accepting any plausible draft token is a different system.
Implementation details also matter when temperature, top-p sampling, tokenizers, or model vocabularies differ.
Treat "lossless" as a claim to verify for the exact engine, method, and sampling path you deploy.
Acceptance rate controls the economics
The draft must be cheap and often right.
If the target accepts most proposed tokens, verification amortizes one target pass across several outputs.
If it rejects early, you paid for the draft and still made little progress.
Measure accepted tokens per verification step, not only the percentage of individual tokens accepted. Early rejection changes how much useful work remains.
Acceptance depends on:
- Similarity between draft and target distributions.
- Prompt domain and output style.
- Sampling temperature and other decoding settings.
- Number of tokens proposed.
- How the draft method uses context.
One configuration can help code completion and lose on creative chat.
Draft models add another model to operate
A small draft model is conceptually simple. It also consumes memory, load time, scheduler attention, and compute.
The draft should be much cheaper than the target while predicting it well enough to earn acceptance.
A model from the same family may share tokenizer and behavior. A much smaller model may run quickly but diverge more often.
Cross-vocabulary methods exist, but translation and constrained sampling add complexity.
The current vLLM speculative decoding guide documents draft models, EAGLE, multi-token prediction, n-gram, suffix, and other methods.
Its guidance is deliberately qualitative. Real gains depend on model family, traffic, hardware, and sampling.
Extra heads avoid a separate draft model
Medusa adds multiple decoding heads that predict future tokens from the target model's hidden states.
This avoids loading a complete second language model.
The heads still need training, compatible checkpoints, tree construction, and verification.
The method can create several candidates for future positions, increasing the chance that verification finds a useful path.
EAGLE predicts future features and uses a lightweight draft component to propose tokens.
These methods change the cost and acceptance tradeoff. They do not make measurement optional.
N-gram and suffix methods use repetition
Many workloads contain repeated text. Code, templates, documents, and long conversations often reuse token sequences already present in context or a cache.
An n-gram proposer searches recent context for a matching prefix and proposes the tokens that followed it.
The proposal is cheap because it does not run another neural model.
It works poorly when the next text is novel. It can work surprisingly well when a response copies, edits, or continues known material.
Suffix-based methods extend this idea with a broader store of prior sequences.
Measure hit rate and accepted length by workload. A global average can hide that only one endpoint benefits.
Low QPS and high QPS are different regimes
At low query volume, one request may leave the GPU underused during decode. Speculation can trade spare compute for lower inter-token latency.
At high query volume, ordinary continuous batching already fills the device with many sequences.
Draft work may then compete with target work and lower total throughput.
This does not mean speculation never helps saturated systems. It means the objective must be explicit.
Test low-load latency, saturated throughput, and goodput under a latency objective.
The continuous batching guide explains why scheduler behavior changes with traffic.
Longer proposals are not always better
Proposing more tokens creates a chance to advance farther.
It also spends more draft work and may reduce the probability that the whole prefix survives.
The best proposal length can change with prompt, model, temperature, and load.
Dynamic methods adjust speculation depth from recent acceptance or model confidence.
Be careful with feedback delay. A controller that reacts to stale acceptance data can oscillate between proposals that are too long and too short.
Log proposal length, accepted length, rejection position, and time spent in draft plus verification.
Benchmark the complete server
Do not benchmark the draft and target in separate scripts and add the times.
Use the serving engine, scheduler, cache allocator, tokenizer, sampling settings, and streaming path that production will use.
Compare against the same engine with speculation disabled.
Keep model revision, precision, context, prompt distribution, output distribution, and arrival process constant.
Report:
- Time to first token and inter-token latency percentiles.
- End-to-end latency and requests per second.
- Output tokens per second and goodput.
- Draft time, verification time, and accepted length.
- GPU memory, power, and failures.
- Output equality or distribution checks required by the method.
Fanout's latency numbers lab helps keep microseconds, milliseconds, and network time in proportion.
A practical adoption sequence
First, establish a stable decode baseline with representative traffic.
Second, try the cheapest compatible proposer, often n-gram or a native multi-token prediction path.
Third, measure acceptance by endpoint and sampling configuration.
Fourth, test a model-based proposer only if the expected latency gain can justify its memory and operating cost.
Fifth, add a runtime switch and safe fallback. A speculative path should be reversible when a model revision or workload changes.
Speculative decoding is a workload optimization, not a badge. Keep it when verified tokens become cheaper or arrive sooner for the users who matter.
The LLM inference roadmap places it after ordinary decode, cache allocation, scheduling, and measurement for a reason.