Measuring Intelligence in the Wild - Arena and the Future of AI Evaluation

Anastasios Angelopoulos, a statistician and co-founder of Arena (formerly LMArena), explains why static benchmarks fall short and how pairwise human-preference data is turned into model leaderboards. The episode examines the statistical machinery behind the rankings, why controlling for response style is necessary yet philosophically difficult, since separating style from substance is an unsolved causal-inference problem, and why users turn out to be good judges of model quality. It also covers pre-release testing of unnamed models, the role of rigorous statistics in AI evaluation and why structured decoding can hurt performance.