The Leaderboard Illusion

The Leaderboard Illusion asks what a live model ranking measures when some providers can privately test many candidates, keep only the strongest score, see more of the prompt distribution, or leave old models in a changing comparison graph. Read it to see why best-of-N selection lifts an observed maximum, how the authors audited private variants and prompt access, and where overlapping intervals, private data, disputed interpretations, and synthetic deprecation studies limit the verdict.

Reading focus: Why selecting the highest of several noisy or genuinely different candidate scores changes what one public leaderboard entry represents. How the audit combines public battle data, identity-based private-model scraping, provider API traffic, controlled submissions, and simulations. Which findings Arena disputed, which policy changes it announced, and what evidence is still needed to separate measurement noise from real model improvement.

NeurIPS 2025. Shivalika Singh et al.. 35 min read, medium difficulty.