Topic: human judgment

Supports multilingual and culturally contextual evaluation of AI systems, including benchmarks and datasets for underrepresented languages and contexts, with attention to the Global South. Proposals may develop responsible evaluation infrastructure, human judgment in evaluation, and methods for assessing AI deployment in emerging markets. Use the ARA proposal template; four pages excluding appendices are encouraged. Decisions are expected in February 2027. Applications close on 4 November 2026 at 11:59pm Pacific Time.

Dimitris Papailiopoulos of Microsoft Research and the University of Wisconsin discusses how coding agents change scientific research. The conversation covers small arithmetic transformers, symbolic mathematical reasoning, interacting agents, and the role of human judgment and verification. It also examines diversity in machine-generated ideas, benchmark and harness overfitting, persistent agent errors, continual learning, world models, and where information theory helps explain AI. Hosted by Ravid Shwartz Ziv and Allen Roush; the full conversation is available on the publisher's YouTube channe