Robust Speech Recognition via Large-Scale Weak Supervision

Whisper trains a standard encoder-decoder Transformer on 680,000 hours of diverse web audio and represents language, transcription, translation, silence, and timestamps with decoder tokens. Read it to see why cross-dataset zero-shot evaluation changes the robustness story, what dataset scale buys, and where long-form generation can still fail.

Reading focus: How one decoder prefix specifies language, task, silence, and timestamp behavior for a fixed audio representation. Why matching two systems on clean LibriSpeech exposes a much larger robustness gap across other speech distributions. How uneven language data, thirty-second windows, decoding heuristics, and hallucinations bound the result.

ICML 2023. Alec Radford et al.. 9 min read, medium difficulty.