Read a speculative decoding acceptance histogram

A speculative decoding acceptance length histogram records how many draft tokens survived each target-model verification step.

If H at index j is 40, then 40 verification steps accepted exactly j draft tokens before the first rejection. The histogram excludes the target model's bonus token in current vLLM metrics.

The mean is useful, but the shape answers better questions: how often drafting fails immediately, how often the full draft survives, and whether another draft position earns enough progress to justify its cost.

Speculative decoding acceptance length histogram

Speculative decoding drafts up to K future tokens and asks the target model to verify them together.

The target accepts the longest valid prefix. Once one draft position fails, later positions in that draft cannot join the accepted prefix even if their token IDs happen to match in isolation.

The original speculative decoding paper shows how verification can preserve the target model's output distribution while reducing serial target calls.

Fanout's speculative decoding guide covers that draft, verify, accept, and reject loop. The histogram measures what the loop did on one request or workload.

Current vLLM per-request acceptance metrics return a dense list with K plus 1 entries.

Index 0 counts steps that accepted no draft tokens. Index K counts steps that accepted the whole draft. Every middle index counts the exact accepted-prefix length.

Recover the basic counts

Let H_j be the count stored at index j, from 0 through K.

The number of speculative verification steps S is the sum of every H_j.

The number of accepted draft tokens A is the sum of j times H_j. This weighted sum ignores the bonus token because the histogram does too.

Under vLLM's convention, mean acceptance length M includes one target-produced bonus token per verification step. Therefore M = 1 + A/S.

For the documented histogram [39, 1, 0, 3], S is 43. A is 1 plus 9, or 10 accepted draft tokens.

Mean acceptance length is 1 + 10/43, or about 1.233 output tokens per target verification step. That matches the example field in the response.

The total progress represented by those steps is S + A, or 53 tokens under this convention.

The target needed 43 serial verification steps for that progress. Ordinary one-token decoding would need roughly 53 target steps, subject to end-of-sequence details.

Mean progress is not draft acceptance rate

vLLM defines draft acceptance rate as A divided by D, where D is the number of draft tokens actually proposed.

In the documented example, D is 129. The rate is 10/129, or about 7.75%.

Mean acceptance length divides by verification steps instead. Draft acceptance rate divides by proposed tokens. They answer different questions and have different denominators.

Do not replace D with K times S when drafting is variable. Structured-output constraints can invalidate proposals, and adaptive methods can propose fewer than K tokens on some steps.

Detailed metrics include per_step_drafted for this reason. The summary also reports num_draft_tokens after removing drafts invalidated by constraints.

Draft waste is D minus A. The example spent work on 119 proposed tokens that did not become accepted draft progress.

That does not prove the configuration is slower. The draft path may be cheap, and target verification may still save enough serial weight reads to win.

Two equal means can describe opposite behavior

Take K = 4 and 100 verification steps.

Workload A has histogram [50, 0, 0, 0, 50]. Half the steps reject at position one, while half accept all four draft tokens.

Its accepted total is 200 and its mean acceptance length is 3, including the bonus token.

Workload B has histogram [0, 0, 100, 0, 0]. Every step accepts exactly two draft tokens.

It also accepts 200 draft tokens and has the same mean of 3. The averages are identical, but the serving behavior is not.

Workload A may mix two prompt domains, one well matched to the drafter and one poorly matched. Workload B is stable but never benefits from positions three and four.

The full-K rate is H_K/S. It is 50% for A and zero for B. The immediate-failure rate is H_0/S. It is 50% for A and zero for B.

Those two rates tell an operator whether to segment traffic or shorten K. The shared mean cannot.

Convert the histogram into position survival

Draft position r contributes a token only when the accepted prefix is at least r tokens long.

The number of steps that accepted position r is the sum of H_j for every j from r through K.

Divide that tail sum by S to get unconditional survival for position r.

For Workload A, every draft position has 50% survival. For Workload B, positions one and two have 100% survival, while positions three and four have zero.

Conditional survival asks a narrower question: among steps that reached position r, how often did that position also survive?

For r greater than one, divide the tail beginning at r by the tail beginning at r minus 1. Position one uses S as its denominator.

This respects prefix dependence. The vLLM Speculators metric implementation uses a cumulative product for the same reason.

Multiplying unrelated per-position marginals can understate or misstate the accepted prefix when positions are correlated.

Choose K from marginal value, not one ratio

The marginal accepted progress from adding draft position r is its survival probability.

If position four survives on only 2% of verification steps, it adds about 0.02 accepted tokens per step before any timing cost is considered.

Measure the extra drafter time and target verification time caused by that position. Keep it only when its saved serial target work is worth more than those additions.

This requires a K sweep. Run K = 1, 2, 3, and higher supported values against the same request order and sampling settings.

For each K, record the histogram, drafted tokens, accepted tokens, drafter time, target verification time, time per output token, and SLO-qualified throughput.

The vLLM server metrics source exposes aggregate counters for drafts, drafted tokens, accepted tokens, and accepted positions.

Use per-request metrics when possible so another client's traffic cannot change the numerator or denominator during the test.

Fanout's goodput comparison explains why the winning K is the one that produces more useful work inside the latency target, not the highest acceptance field.

Segment the histogram before tuning

A pooled histogram can combine code, prose, translation, retrieval-grounded answers, and structured outputs into one shape.

Draft quality often differs across those workloads. N-gram drafting may excel when an answer copies prompt text and fail on novel prose.

Segment by drafter method, target and draft checkpoint, sampling configuration, prompt domain, context bucket, output bucket, and active batch size.

Also segment by K when dynamic speculation changes the maximum draft length. Histograms with different lengths cannot be added index by index without labeling the K used.

For adaptive drafting, use per_step_drafted beside per_step_accepted. A step that accepted two of two proposals is different from one that accepted two of four.

Current vLLM reports the per-request fields only for single-sequence requests. Its documentation says the field is null when n is greater than one.

Streaming clients receive the metrics on the final usage chunk, so usage reporting must be enabled and the client must retain that last event.

Keep timing beside acceptance

An acceptance histogram contains no time axis. It cannot tell whether the drafter was slow, verification shapes missed CUDA graphs, or a larger batch made parallel verification expensive.

The Modular speculative decoding guide warns that draft-model memory and wasted compute can erase the gain even when the algorithm is correct.

Measure accepted throughput as A divided by wall time, but also include the bonus-token progress when comparing end-to-end output speed.

Record target verification duration per step. A longer accepted prefix helps only if verifying K positions remains cheaper than the serial target calls it replaces.

Record inter-token latency too. Speculative decoding can produce bursts of accepted tokens followed by a longer verification gap.

Fanout's smooth goodput guide separates aggregate useful output from the cadence a streaming user sees.

Compare against a server launched without speculative decoding. Selecting K = 0 inside a dynamic speculative engine may leave graph, scheduler, or drafter-state costs in place.

A practical acceptance report

Start the report with K, S, D, A, and the histogram. State whether the histogram excludes or includes the bonus token.

Calculate mean acceptance length as 1 + A/S and draft acceptance rate as A/D. Report both labels and both denominators.

Add immediate-failure rate H_0/S and full-draft rate H_K/S. Then list survival probability for each draft position.

Show the histogram separately for the workload segments that make different product decisions. A single fleet-wide bar can hide one domain that should disable speculation.

Beside each segment, report no-speculation TPOT, speculative TPOT, p95 inter-token latency, drafter time, verification time, and memory overhead.

Choose K from the last position whose marginal survival still improves the target service metric. Re-run after changing the checkpoint, sampling, schema, or batch policy.

The histogram's weighted sum gives the mean. Its tail counts show whether to shorten K, segment the traffic, or keep the current drafter.