---
title: "Fast Inference from Transformers via Speculative Decoding"
description: "Speculative decoding makes autoregressive generation faster without changing the target model's output distribution. Read it to see how a small draft model, parallel verification, and a corrected sampling rule turn extra compute into lower decoding latency without retraining the models."
canonical_url: "https://fanout.sh/daily/2026-07-27-speculative-decoding"
md_url: "https://fanout.sh/daily/2026-07-27-speculative-decoding.md"
last_updated: "2026-07-27"
access: "public"
---

# Fast Inference from Transformers via Speculative Decoding

Speculative decoding makes autoregressive generation faster without changing the target model's output distribution. Read it to see how a small draft model, parallel verification, and a corrected sampling rule turn extra compute into lower decoding latency without retraining the models.

## Paper details

- Authors: Leviathan, Kalman, and Matias

- Venue: ICML 2023

- Track: Inference

- Difficulty: Medium

- Reading time: 40 min

- Original paper: https://proceedings.mlr.press/v202/leviathan23a.html

## What you will learn

- How a smaller draft model proposes several likely tokens before the larger target model runs.

- How the target model verifies a block of draft tokens in parallel instead of decoding every token serially.

- Why the acceptance and correction rule preserves the target model's output distribution while enabling practical speedups.

## Continue exploring

[Explore Speculative Decoding](https://fanout.sh/knowledge-graph?node=ai-research%3Aconcept%3Aspeculative-decoding): Open Fanout's speculative-decoding node and connect the paper to draft models, parallel verification, sampling, and low-latency LLM serving.

---
This representation contains public Fanout content only. Protected Pro lessons, account data, billing, checkout, and pricing are not included.

Browse the public content map: https://fanout.sh/sitemap.md