Topic: Replication

Fidelity and Replication: Modelling the Impact of Protocol Deviations on Effect Size

Cognitive science and cognitive neuroscience researchers have agreed that the replication of findings is important for establishing which ideas (or theories) are integral to the study of cognition across the lifespan. Recently, high-profile papers have called into question findings that were once thought to be unassailable. Much attention has been paid to how p-hacking, publication bias, and sample size are responsible for failed replications. However, much less attention has been paid to the fidelity by which researchers enact study protocols. Researchers conducting education or clinical trials are aware of the importance in fidelity – or the extent to which the protocols are delivered in the same way across participants. Nevertheless, this idea has not been applied to cognitive contexts. This seminar discusses factors that impact the replicability of findings alongside recent models suggesting that even small fidelity deviations have real impacts on the data collected.

Neuroimaging reproducibility - pain points and roadmap for solid and reusable results

There is a growing body of evidence that reproducibility or replication is low in neuroscience and in neuroimaging in particular, but the factors affecting studies solidity are still generally poorly understood, and the solutions are not clearly exposed to the neuroimaging scientific community. In this talk, I will review the key factors contributing to irreproducible results in neuroimaging specifically in the context of explanatory or prediction studies and propose a series of practical steps to improve the neuroimaging (and neuroscience) results robustness and re-usability.

Lessons from the credibility revolution – social thermoregulation as a case study

The goal of this talk is to first provide a realization of why the replication crisis is omnipresent and then point to several tools via which the listener can improve their own work. To do so, I will go through our own work on social thermoregulation, point out why I thought changes were necessary, discuss which shortcomings we have in our own work, which measures we have taken to reduce those shortcomings, which tools we have relied on to do so, and which steps I believe we still need to make. Specifically, I will go through the following points: Major replication failures and data fabrication in the field of psychology; Replication failures of social thermoregulation studies; Realization that many of our studies were underpowered; Realization that many of our studies were very narrow in scope (i.e., in undergraduate students and mostly in EU/US); Realization that a lot of our measures were not independently validated. I will show these for our own work (but will also show why, via a meta-analysis, we have enough confidence to proceed with social thermoregulation research). Throughout the talk I will point you to the following tools that facilitate our work: Templates for exploratory and confirmatory research and for meta-analyses (developed for our work, but easily adaptable for other programs). I will also show you how to fork our templates; A lab philosophy; A research milestones sheet for collaborations and overviews; Excel sheet for contributorship; A tutorial for exploratory research; I would recommend listeners to read through this chapter before the talk (I will repeat a lot of that work, but I will go into greater depth). own work. To do so, I will go through our own work on social thermoregulation, point out why I thought changes were necessary, discuss which shortcomings we have in our own work, which measures we have taken to reduce those shortcomings, which tools we have relied on to do so, and which steps I believe we still need to make.

Adventures in DNA replication using single molecule biophysics

Searching for the Super-Searchers

A striking range of individual differences has been reported in a variety of visual search tasks, which naturally leads to the idea that some people are better at finding things than others. However, this conclusion appears to be premature. We carried out a replication of three recent visual search experiments and found that each task showed a wide range of individual differences as predicted, and observed good test-retest reliability in all three. However, performance on any one task was not correlated with the performance in the others: participants who naturally adopt efficient search strategies in one paradigm may perform at chance in another! Furthermore, we also show that behaviour in different versions of the same paradigm can be radically different: When simple line segments are used for search items, a large range of search strategies are found. If we instead use more complex search items, all our participants effortlessly adopt an optimal strategy. These results suggest search strategies are stable over time, but context-specific. To understand visual search we, therefore, need to account not only for differences between individuals but also how individuals interact with the search task and context.

The recent history of the replication crisis in psychology & how Open Science can be part of the solution

In recent years, more and more evidence has accumulated showing that many studies in psychological research cannot be replicated, effects are often overestimated, and little is publicly known about unsuccessful studies. What are the mechanisms behind this crisis? In this talk, I will explain how we got there and why it is still difficult to break free from the current system. I will further explain which role Open Science plays within the replication crisis and how it can help to improve science. This might sound like a pessimistic, negative talk, but I will end it on a positive note, I promise!

“Super Spreaders in the Corona Epidemics”

Recently a powerful example of a replicating nano-machine entered our society. In principle, it’s just a normal disease, that one attempts to model with 3 or 4 simple coupled equations with 2 important parameters: a timescale, and a replication factor (the famous R0). Then one tries to guess how changes in society change R0 and perhaps adopt some more or less strong lock-down measures. However, this virus has more “personality” than that. It behaves differently in different persons, and persons behave differently. Presumably, only a few of us infect a lot, while most do not infect so much. This assumption is supported by the observation that couples living together only infect each other with about 15 percent probability, indicating that most infected people are not really infectious. I will discuss this and other aspects of Covid-19 in the perspective of models that describe heterogeneous individuals in a society. In particular, we suggest that limiting superspreading opportunities is a cost-effective strategy to mitigate Covid-19.

Measuring transcription at a single gene copy reveals hidden drivers of bacterial individuality

Single-cell measurements of mRNA copy numbers inform our understanding of stochastic gene expression, but these measurements coarse-grain over the individual copies of the gene, where transcription and its regulation take place stochastically. We recently combined single-molecule quantification of mRNA and gene loci to measure the transcriptional activity of an endogenous gene in individual Escherichia coli bacteria. When interpreted using a theoretical model for mRNA dynamics, the single-cell data allowed us to obtain the probabilistic rates of promoter switching, transcription initiation and elongation, mRNA release and degradation. Unexpectedly, we found that gene activity can be strongly coupled to the transcriptional state of another copy of the same gene present in the cell, and to the event of gene replication during the bacterial cell cycle. These gene-copy and cell-cycle correlations demonstrate the limits of mapping whole-cell mRNA numbers to the underlying stochastic gene activity and highlight the contribution of previously hidden variables to the observed population heterogeneity.