publication bias

Fidelity and Replication: Modelling the Impact of Protocol Deviations on Effect Size

Cognitive science and cognitive neuroscience researchers have agreed that the replication of findings is important for establishing which ideas (or theories) are integral to the study of cognition across the lifespan. Recently, high-profile papers have called into question findings that were once thought to be unassailable. Much attention has been paid to how p-hacking, publication bias, and sample size are responsible for failed replications. However, much less attention has been paid to the fidelity by which researchers enact study protocols. Researchers conducting education or clinical trials are aware of the importance in fidelity – or the extent to which the protocols are delivered in the same way across participants. Nevertheless, this idea has not been applied to cognitive contexts. This seminar discusses factors that impact the replicability of findings alongside recent models suggesting that even small fidelity deviations have real impacts on the data collected.