Workshop
Effective and Efficient Evaluation for AI Scientists
Bang Liu, Fan Zhou, Perouz Taslakian and 3 more
About
This AAAI-27 workshop asks how the outputs of autonomous AI Scientists, systems that generate hypotheses, design experiments, run code and draft papers, should be evaluated, and how the evaluation itself can be trusted. It treats the measurement instrument as the object of study across three questions: what to measure (operational definitions of novelty, correctness, rigour and significance and their construct validity; ground truth stronger than reviewer opinion, such as held-out empirical outcomes and laboratory confirmation; process-level evaluation from agent traces), how to measure it (end-to-end and stage-wise benchmarks for research agents, contamination and leakage, LLM-as-a-judge reliability and calibration with cost-normalised reporting) and how good the instrument is (reliability versus validity, statistical power, sample-efficient evaluation, ranking sensitivity to judge, prompt and rubric). Expert elicitation, reproducibility infrastructure, hazard and dual-use assessment for autonomous chemistry and biology, and Goodharting are also in scope. The half-day programme has invited talks, 18 oral slots, posters, a panel and the AI Scientist Evaluation Challenge, a shared task. Full, position, resource and reproduction or negative-result papers are due by 06:00 UTC on 20 November 2026 and are presented in person. Attendance is capped at 45 to 65 participants, by invitation on the basis of submissions, with further AAAI-27 registrants admitted as capacity permits.
Topics
Organisers
- Bang Liu, Université de Montréal / Mila
- Fan Zhou, Shanghai University of Finance and Economics
- Perouz Taslakian, ServiceNow AI Research / McGill University / Mila
- Samira Ebrahimi Kahou, University of Calgary
- Steve Drew, University of Calgary
- Tianyu Shi, McGill University
Guest speakers
- Irina Rish, Université de Montréal and Mila
- Hongtu Zhu, University of North Carolina at Chapel Hill
- Mengdi Wang, Princeton University
- Phoenix Peng, Gestala
- Xiaowu Dai, University of California, Los Angeles
- Yifan Hu, Rutgers University
- Zhijing Jin, University of Toronto
Details
Location
Palais des congrès de Montréal, Montréal, Canada
Organisations
Association for the Advancement of Artificial Intelligence (AAAI)
Host