Let's Verify Step by Step

Let's Verify Step by Step compares reward models trained on a final answer with reward models trained to judge every reasoning step. Read it to see why first-error labels improve credit assignment, how best-of-N exposes verifier quality without changing the generator, and why the headline result needs a controlled companion experiment.

Reading focus: How process supervision preserves a valid prefix while locating the first broken reasoning step. Why product scoring turns many step judgments into one candidate rank for best-of-N selection. Which results isolate supervision form and which remain confounded by different training sets.

ICLR 2024. Hunter Lightman et al.. 32 min read, medium difficulty.