COVID-19 non-pharmaceutical intervention (NPI) policies have been one of the most important and contentious decisions of our time. Beyond even the "normal" inherent difficulties in impact evaluation with observational data, COVID-19 NPI policy evaluation is complicated by additional challenges related to infectious disease dynamics and lags, lack of direct observation of key outcomes, and a multiplicity of interventions occurring on an accelerated time scale. Randomized controlled trials also suffer from what is feasible and ethical to randomize as well as the sheer scale, scope, time, and resources required for an NPI trial to be informative (or at least not misinformative). In this talk, Dr. Haber will discuss the challenges in generating useful evidence for COVID-19 NPIs, the landscape of the literature, and highlight key controversies in several high profile studies over the course of the pandemic. Chasing after unknowables poses major problems for the metascience/replicability movement, institutional research science, and decision makers. If the only choices for informing an important topic are "weak study design" vs "do nothing," when is "do nothing" the best choice?

Progress in psychology has been frustrated by challenges concerning replicability, generalizability, strategy selection, inferential reproducibility, and computational reproducibility. Although often discussed separately, I argue that these five challenges share a common cause: insufficient investment of resources into the typical psychology study. I further suggest that big team science can help address these challenges by allowing researchers to pool their resources to efficiently and drastically increase the amount of resources available for a single study. However, the current incentives, infrastructure, and institutions in academic science have all developed under the assumption that science is conducted by solo Principal Investigators and their dependent trainees. These barriers must be overcome if big team science is to be sustainable. Big team science likely also carries unique risks, such as the potential for big team science institutions to monopolize power, become overly conservative, make mistakes at a grand scale, or fail entirely due to mismanagement and a lack of financial sustainability. I illustrate the promise, barriers, and risks of big team science with the experiences of the Psychological Science Accelerator, a global research network of over 1400 members from 70+ countries.