--- title: "Structured output speculative decoding acceptance" description: "Separate grammar-invalid drafts from target-model rejections so structured-output acceptance rates keep a stable, useful denominator." canonical_url: "https://fanout.sh/blog/structured-output-speculative-decoding-acceptance" md_url: "https://fanout.sh/blog/structured-output-speculative-decoding-acceptance.md" last_updated: "2026-10-09" access: "public" --- # Structured output speculative decoding acceptance Separate grammar-invalid drafts from target-model rejections so structured-output acceptance rates keep a stable, useful denominator. - Author: Suraj Gaud - Published: 2026-10-09 - Track: Inference engineering - Access: Public - Tags: speculative decoding, structured output, acceptance rate, vLLM, constrained decoding, JSON schema, inference metrics Structured output speculative decoding acceptance can rise for two different reasons. A grammar can make the next token easier to predict, or it can remove invalid draft tokens before the target-model acceptance rate is calculated. Those effects should not share one unnamed percentage. Track raw draft proposals, grammar-eligible proposals, and target-accepted proposals as separate counters. Current vLLM per-request metrics make this distinction explicit in one place: num_draft_tokens subtracts drafts invalidated by structured-output constraints. ## Structured output speculative decoding acceptance Speculative decoding asks a cheap drafter to propose several future tokens. The target model scores them in parallel and accepts the valid prefix under the chosen verification rule. The original[speculative decoding paper](https://arxiv.org/abs/2211.17192)shows how this can preserve the target distribution while replacing several serial target runs with one parallel verification pass. Structured output adds another gate. A JSON schema, regex, choice set, or grammar restricts which tokens may appear at the current parser state. vLLM's[structured-output documentation](https://github.com/vllm-project/vllm/blob/main/docs/features/structured_outputs.md)lists those constraint types and the backends that compile them into token masks. A draft token can now fail before target verification because the grammar forbids it. A grammar-legal draft can still fail because the target distribution rejects it. Fanout's[speculative acceptance histogram guide](/blog/speculative-decoding-acceptance-length-histogram)covers accepted-prefix length by step. Structured output adds a second reason that a proposed prefix can shorten. ## Name the three counters Let R be raw draft tokens produced by the drafter before grammar validation. Let I be drafts invalidated by the structured-output constraint. The eligible draft count is D = R minus I. Let A be eligible draft tokens accepted by the target verifier. Three ratios now answer different questions. - Grammar survival rate is D divided by R. It measures how often the drafter stays inside the active constraint. - Conditional target acceptance is A divided by D. It measures agreement after invalid drafts are removed. - End-to-end draft yield is A divided by R. It measures accepted work against everything the drafter produced. The[vLLM acceptance-metrics contract](https://docs.vllm.ai/en/latest/features/speculative_decoding/acceptance_metrics/)defines draft_acceptance_rate as A divided by D. Its num_draft_tokens field is D, not R, because the implementation subtracts drafts invalidated by structured-output constraints. The conditional metric is useful. It becomes misleading when a dashboard labels it as the share of all generated draft tokens that survived. ## Work a denominator example Suppose a drafter emits R = 1,000 tokens during JSON-schema generation. Grammar validation invalidates I = 200, leaving D = 800 eligible tokens. The target accepts A = 600. vLLM's documented draft acceptance rate is 600 divided by 800, or 75 percent. Grammar survival is 800 divided by 1,000, or 80 percent. End-to-end draft yield is 600 divided by 1,000, or 60 percent. Now compare an unconstrained run with R = D = 1,000 and A = 600. Its reported acceptance rate is 60 percent even though both runs turn 600 raw proposals into accepted tokens. The structured run's 75 percent is useful evidence that eligible drafts align better with the target. It does not prove the drafter delivered more accepted work per raw proposal. Compare one more run with R = 1,000, I = 0, and A = 750. It also reports 75 percent, but it accepts 150 more drafted tokens than the first run. A stable evaluation therefore needs A/D and A/R. Without R or I, conditional acceptance cannot reveal how much draft work the grammar discarded. ## Mean acceptance length uses another denominator Acceptance rate is not the same as mean acceptance length. The first counts accepted draft tokens per eligible proposal. The second counts emitted tokens per verification step. vLLM defines mean acceptance length as 1 plus A divided by the number of speculative steps. The added one is the target model's bonus token. If 100 steps accept 180 draft tokens, mean acceptance length is 2.8 tokens per step. This remains true whether the configured maximum draft length is three or eight. The acceptance histogram records how many steps accepted zero, one, or more draft tokens. It excludes the bonus token and has one bucket for every possible accepted draft length. With detailed metrics, per_step_drafted records the effective proposal length and per_step_accepted records the accepted length for each verification step. Use those arrays to distinguish a short eligible draft from a long eligible draft rejected at position one. Both can contribute one accepted token but imply different tuning work. Fanout's[dynamic speculation guide](/blog/dynamic-speculative-decoding-batch-size-thresholds)explains why proposal length and batch shape must be logged beside acceptance. ## Constraints can help and filter at once Page-one explainers often say code, JSON, and other structured text produce high acceptance because their token distributions are predictable. The claim is plausible but incomplete.[Online Speculative Decoding](https://proceedings.mlr.press/v235/liu24y.html)shows that agreement depends on the gap between draft and target distributions and on the query distribution. A grammar can narrow the legal token set, which may make draft and target choices align more often. It can also expose a drafter that repeatedly proposes illegal punctuation, keys, or closing tokens. Those are different interventions. Better prompting or draft training can raise grammar survival. A closer draft-target pair can raise conditional target acceptance. Changing the schema can move both. A fixed enum creates fewer legal branches than an object with free-form strings, so one workload-level acceptance number hides the constraint's difficulty. Slice the metrics by schema family, grammar backend, drafter method, target model, sampling parameters, and reasoning mode. Do not compare structured and unstructured traffic by acceptance rate alone. Their D denominators may describe different fractions of the drafter's raw work. ## Reasoning boundaries complicate the window Some models produce unconstrained reasoning before a constrained final answer. The constraint can begin in the middle of one speculative draft window. The vLLM[StructuredOutputManager RFC](https://github.com/vllm-project/vllm/issues/48197)describes the required sequence. The manager preserves the unconstrained prefix, validates the constrained suffix, builds the correct masks, and advances grammar state only with verified constrained tokens. Rejected drafts must not advance the grammar state. Drafts may temporarily advance a copy of that state to construct masks for later positions in the same speculative window. A counter is trustworthy only if validation, verification, and grammar advancement agree about which token positions were constrained. Pin the serving version when relying on the experimental per-request metric shape. Test a reasoning boundary that lands before, inside, and after the configured draft window. ## Build a structured-output scorecard Start with correctness: schema-valid completion rate, parser errors, truncated responses, and server errors. A fast acceptance number cannot compensate for an invalid response. Then record R, I, D, A, speculative steps, accepted-length histogram, and bonus tokens. If the serving response exposes only D and A, add raw proposal and invalidation counters near grammar validation. Calculate grammar survival, conditional target acceptance, end-to-end draft yield, and mean acceptance length from those counters. Add latency and cost: drafter time, target verification time, mask-generation time, tokens per second, inter-token latency, and GPU utilization. Replay the same prompts with constraints on and off when the output contract permits both. Keep temperature, seed, draft length, concurrency, and model pair fixed. Finally, bucket by constraint shape. Small enums, nested JSON, regex-heavy strings, and mixed reasoning-plus-JSON responses exercise different parser states and legal-token sets. Structured output speculative decoding acceptance needs two rejection labels. Grammar invalidation measures whether the draft stays legal. Target rejection measures whether the legal draft matches the target. Report A/D and A/R together. The first tunes agreement after constraints. The second tells whether every unit of draft work is buying accepted output. --- This representation contains public Fanout content only. Protected Pro lessons, account data, billing, checkout, and pricing are not included. Browse the public content map: https://fanout.sh/sitemap.md