Fast Inference from Transformers via Speculative…

Speculative decoding makes autoregressive generation faster without changing the target model's output distribution. Read it to see how a small draft model, parallel verification, and a corrected sampling rule turn extra compute into lower decoding latency without retraining the models.

Reading focus: How a smaller draft model proposes several likely tokens before the larger target model runs. How the target model verifies a block of draft tokens in parallel instead of decoding every token serially. Why the acceptance and correction rule preserves the target model's output distribution while enabling practical speedups.

ICML 2023. Leviathan, Kalman, and Matias. 40 min read, medium difficulty.