Why AI Benchmarks Are Lying to You - with Wenhu Chen (Meta/University of Waterloo)

Wenhu Chen, research scientist at Meta and assistant professor at the University of Waterloo and the creator of MMLU-Pro and MMMU, examines how much benchmark gains reflect real progress. Benchmarks act as the field's loss function, so a poorly designed one sends researchers hill-climbing in the wrong direction. He discusses contamination, saturation that turns frontier benchmarks into unit tests, and preference arenas that mostly reward tone and length, and argues for evaluation in settings models have never seen. He also describes ClawBench, which sends agents to more than 140 real websites to complete everyday tasks, and why the best models finish only about a third of them.