Sharpness-Aware Minimization for Efficiently Improving Generalization

Sharpness-Aware Minimization trains for a low-loss neighborhood rather than one low-loss point. Follow the two-pass update, the rho-sized adversarial weight probe, compute-aware vision results, official implementation, and the later theory that narrows the flatness story.

Reading focus: How SAM turns a worst-case loss inside a small weight-space neighborhood into a practical two-gradient update. Why the first pass only locates a high-loss perturbation while the second gradient updates the original weights. What the reported CIFAR, ImageNet, finetuning, and noisy-label gains establish, and where extra compute, rho tuning, parameterization, and later theory limit the claim.

ICLR 2021. Pierre Foret, Ariel Kleiner, Hossein Mobahi, and Behnam Neyshabur. 10 min read, medium difficulty.