Topic: Reproducibility

Introduction to protocols.io: Scientific collaboration through open protocols

Research articles and laboratory protocol organization often lack detailed instructions for replicating experiments. protocols.io is an open-access platform where researchers collaboratively create dynamic, interactive, step-by-step protocols that can be executed on mobile devices or the web. Researchers can easily and efficiently share protocols with colleagues, collaborators, the scientific community, or make them public. Real-time communication and interaction keep protocols up to date. Public protocols receive a DOI and enable open communication with authors and researchers to foster efficient experimentation and reproducibility.

Introduction to protocols.io: Scientific collaboration through open protocols

Research articles and laboratory protocol organization often lack detailed instructions for replicating experiments. protocols.io is an open-access platform where researchers collaboratively create dynamic, interactive, step-by-step protocols that can be executed on mobile devices or the web. Researchers can easily and efficiently share protocols with colleagues, collaborators, the scientific community, or make them public. Real-time communication and interaction keep protocols up to date. Public protocols receive a DOI and enable open communication with authors and researchers to foster efficient experimentation and reproducibility.

OpenNeuro FitLins GLM: An Accessible, Semi-Automated Pipeline for OpenNeuro Task fMRI Analysis

In this talk, I will discuss the OpenNeuro Fitlins GLM package and provide an illustration of the analytic workflow. OpenNeuro FitLins GLM is a semi-automated pipeline that reduces barriers to analyzing task-based fMRI data from OpenNeuro's 600+ task datasets. Created for psychology, psychiatry and cognitive neuroscience researchers without extensive computational expertise, this tool automates what is largely a manual process and compilation of in-house scripts for data retrieval, validation, quality control, statistical modeling and reporting that, in some cases, may require weeks of effort. The workflow abides by open-science practices, enhancing reproducibility and incorporates community feedback for model improvement. The pipeline integrates BIDS-compliant datasets and fMRIPrep preprocessed derivatives, and dynamically creates BIDS Statistical Model specifications (with Fitlins) to perform common mass univariate [GLM] analyses. To enhance and standardize reporting, it generates comprehensive reports which includes design matrices, statistical maps and COBIDAS-aligned reporting that is fully reproducible from the model specifications and derivatives. OpenNeuro Fitlins GLM has been tested on over 30 datasets spanning 50+ unique fMRI tasks (e.g., working memory, social processing, emotion regulation, decision-making, motor paradigms), reducing analysis times from weeks to hours when using high-performance computers, thereby enabling researchers to conduct robust single-study, meta- and mega-analyses of task fMRI data with significantly improved accessibility, standardized reporting and reproducibility.

Recent views on pre-registration

A discussion on some recent perspectives on pre-registration, which has become a growing trend in the past few years. This is not just limited to neuroimaging, and it applies to most scientific fields. We will start with this overview editorial by Simmons et al. (2021): https://faculty.wharton.upenn.edu/wp-content/uploads/2016/11/34-Simmons-Nelson-Simonsohn-2021a.pdf, and also talk about a more critical perspective by Pham & Oh (2021): https://www.researchgate.net/profile/Michel-Pham/publication/349545600_Preregistration_Is_Neither_Sufficient_nor_Necessary_for_Good_Science/links/60fb311e2bf3553b29096aa7/Preregistration-Is-Neither-Sufficient-nor-Necessary-for-Good-Science.pdf. I would like us to discuss the pros and cons of pre-registration, and if we have time, I may do a demonstration of how to perform a pre-registration through the Open Science Framework.

Towards open meta-research in neuroimaging

When meta-research (research on research) makes an observation or points out a problem (such as a flaw in methodology), the project should be repeated later to determine whether the problem remains. For this we need meta-research that is reproducible and updatable, or living meta-research. In this talk, we introduce the concept of living meta-research, examine prequels to this idea, and point towards standards and technologies that could assist researchers in doing living meta-research. We introduce technologies like natural language processing, which can help with automation of meta-research, which in turn will make the research easier to reproduce/update. Further, we showcase our open-source litmining ecosystem, which includes pubget (for downloading full-text journal articles), labelbuddy (for manually extracting information), and pubextract (for automatically extracting information). With these tools, you can simplify the tedious data collection and information extraction steps in meta-research, and then focus on analyzing the text. We will then describe some living meta-research projects to illustrate the use of these tools. For example, we’ll show how we used GPT along with our tools to extract information about study participants. Essentially, this talk will introduce you to the concept of meta-research, some tools for doing meta-research, and some examples. Particularly, we want you to take away the fact that there are many interesting open questions in meta-research, and you can easily learn the tools to answer them. Check out our tools at https://litmining.github.io/

NII Methods (journal club): NeuroQuery, comprehensive meta-analysis of human brain mapping

We will discuss a recent paper by Taylor et al. (2023): https://www.sciencedirect.com/science/article/pii/S1053811923002896. They discuss the merits of highlighting results instead of hiding them; that is, clearly marking which voxels and clusters pass a given significance threshold, but still highlighting sub-threshold results, with opacity proportional to the strength of the effect. They use this to illustrate how there in fact may be more agreement between researchers than previously thought, using the NARPS dataset as an example. By adopting a continuous, "highlighted" approach, it becomes clear that the majority of effects are in the same location and that the effect size is in the same direction, compared to an approach that only permits rejecting or not rejecting the null hypothesis. We will also talk about the implications of this approach for creating figures, detecting artifacts, and aiding reproducibility.

Research Data Management in neuroimaging

This set of short webinars will provide neuroscience researchers working in a neuroimaging setting with practical tips on strengthening credibility at different stages of the research project. Each webinar will be hosted by Cassandra Gould Van Praag from the Wellcome Centre for Integrative Neuroimaging.

Data privacy for neuroimaging

This set of short webinars will provide neuroscience researchers working in a neuroimaging setting with practical tips on strengthening credibility at different stages of the research project. Each webinar will be hosted by Cassandra Gould Van Praag from the Wellcome Centre for Integrative Neuroimaging.

Preregistration in neuroimaging

This set of short webinars will provide neuroscience researchers working in a neuroimaging setting with practical tips on strengthening credibility at different stages of the research project. Each webinar will be hosted by Cassandra Gould Van Praag from the Wellcome Centre for Integrative Neuroimaging.

Toward an open science ecosystem for neuroimaging

It is now widely accepted that openness and transparency are keys to improving the reproducibility of scientific research, but many challenges remain to adoption of these practices. I will discuss the growth of an ecosystem for open science within the field of neuroimaging, focusing on platforms for open data sharing and open source tools for reproducible data analysis. I will also discuss the role of the Brain Imaging Data Structure (BIDS), a community standard for data organization, in enabling this open science ecosystem, and will outline the scientific impacts of these resources.

Sharing data from your in vivo studies

Handling data in your in vivo studies

Preregistering your in vivo studies

Pynapple: a light-weight python package for neural data analysis - webinar + tutorial

In systems neuroscience, datasets are multimodal and include data-streams of various origins: multichannel electrophysiology, 1- or 2-p calcium imaging, behavior, etc. Often, the exact nature of data streams are unique to each lab, if not each project. Analyzing these datasets in an efficient and open way is crucial for collaboration and reproducibility. In this combined webinar and tutorial, Adrien Peyrache and Guillaume Viejo will present Pynapple, a Python-based data analysis pipeline for systems neuroscience. Designed for flexibility and versatility, Pynapple allows users to perform cross-modal neural data analysis via a common programming approach which facilitates easy sharing of both analysis code and data.

Pynapple: a light-weight python package for neural data analysis - webinar + tutorial

In systems neuroscience, datasets are multimodal and include data-streams of various origins: multichannel electrophysiology, 1- or 2-p calcium imaging, behavior, etc. Often, the exact nature of data streams are unique to each lab, if not each project. Analyzing these datasets in an efficient and open way is crucial for collaboration and reproducibility. In this combined webinar and tutorial, Adrien Peyrache and Guillaume Viejo will present Pynapple, a Python-based data analysis pipeline for systems neuroscience. Designed for flexibility and versatility, Pynapple allows users to perform cross-modal neural data analysis via a common programming approach which facilitates easy sharing of both analysis code and data.

Retinal neurogenesis and lamination: What to become, where to become it and how to move from there!

The vertebrate retina is an important outpost of the central nervous system, responsible for the perception and transmission of visual information. It consists of five different types of neurons that reproducibly laminate into three layers, a process of crucial importance for the organ’s function. Unsurprisingly, impaired fate decisions as well as impaired neuronal migrations and lamination lead to impaired retinal function. However, how processes are coordinated at the cellular and tissue level and how variable or robust retinal formation is, is currently still underexplored. In my lab, we aim to shed light on these questions from different angles, studying on the one hand differentiation phenomena and their variability and on the other hand the downstream migration and lamination phenomena. We use zebrafish as our main model system due to its excellent possibilities for live imaging and quantitative developmental biology. More recently we also started to use human retinal organoids as a comparative system. We further employ cross disciplinary approaches to address these issues combining work of cell and developmental biology, biomechanics, theory and computer science. Together, this allows us to integrate cell with tissue-wide phenomena and generate an appreciation of the reproducibility and variability of events.

How evidence synthesis can boost in vivo credibility

As part of the BNA's ongoing Credibility in Neuroscience work, this series of three short webinars will provide neuroscience researchers working in an in vivo setting with tips on how to improve the credibility of their work. Each webinar will be hosted by Emily Sena, member of the BNA's Credibility Advisory Board, with the opportunity for questions.

Embracing variation to boost reproducibility

As part of the BNA's ongoing Credibility in Neuroscience work, this series of three short webinars will provide neuroscience researchers working in an in vivo setting with tips on how to improve the credibility of their work. Each webinar will be hosted by Emily Sena, member of the BNA's Credibility Advisory Board, with the opportunity for questions.

Improving reliability through design and reporting

As part of the BNA's ongoing Credibility in Neuroscience work, this series of three short webinars will provide neuroscience researchers working in an in vivo setting with tips on how to improve the credibility of their work. Each webinar will be hosted by Emily Sena, member of the BNA's Credibility Advisory Board, with the opportunity for questions.

ReproNim: Towards a culture of more reproducible neuroimaging research

Given the intrinsically large and complex data sets collected in neuroimaging research, coupled with the extensive array of shared data and tools amassed in the research community, ReproNim seeks to lower the barriers for efficient: use of data; description of data and process; use of standards and best practices; sharing; and subsequent reuse of the collective ‘big’ data. Aggregation of data and reuse of analytic methods have become critical in addressing concerns about the replicability and power of many of today’s neuroimaging studies.

Autopilot v0.4.0 - Distributing development of a distributed experimental framework

Autopilot is a Python framework for performing complex behavioral neuroscience experiments by coordinating a swarm of Raspberry Pis. It was designed to not only give researchers a tool that allows them to perform the hardware-intensive experiments necessary for the next generation of naturalistic neuroscientific observation, but also to make it easier for scientists to be good stewards of the human knowledge project. Specifically, we designed Autopilot as a framework that lets its users contribute their technical expertise to a cumulative library of hardware interfaces and experimental designs, and produce data that is clean at the time of acquisition to lower barriers to open scientific practices. As autopilot matures, we have been progressively making these aspirations a reality. Currently we are preparing the release of Autopilot v0.4.0, which will include a new plugin system and wiki that makes use of semantic web technology to make a technical and contextual knowledge repository. By combining human readable text and semantic annotations in a wiki that makes contribution as easy as possible, we intend to make a communal knowledge system that gives a mechanism for sharing the contextual technical knowledge that is always excluded from methods sections, but is nonetheless necessary to perform cutting-edge experiments. By integrating it with Autopilot, we hope to make a first of its kind system that allows researchers to fluidly blend technical knowledge and open source hardware designs with the software necessary to use them. Reciprocally, we also hope that this system will support a kind of deep provenance that makes abstract "custom apparatus" statements in methods sections obsolete, allowing the scientific community to losslessly and effortlessly trace a dataset back to the code and hardware designs needed to replicate it. I will describe the basic architecture of Autopilot, recent work on its community contribution ecosystem, and the vision for the future of its development.

How we can make 3D models more reproducible

Reproducible research using stem cell derived neurons and organoids

Neuroimaging reproducibility - pain points and roadmap for solid and reusable results

There is a growing body of evidence that reproducibility or replication is low in neuroscience and in neuroimaging in particular, but the factors affecting studies solidity are still generally poorly understood, and the solutions are not clearly exposed to the neuroimaging scientific community. In this talk, I will review the key factors contributing to irreproducible results in neuroimaging specifically in the context of explanatory or prediction studies and propose a series of practical steps to improve the neuroimaging (and neuroscience) results robustness and re-usability.

A discussion on the necessity for Open Source Hardware in neuroscience research

Research tools are paramount for scientific development, they enable researchers to observe and manipulate natural phenomena, learn their principles, make predictions and develop new technologies, treatments and improve living standards. Due to their costs and the geographical distribution of manufacturing companies access to them is not widely available, hindering the pace of research, the ability of many communities to contribute to science and education and reap its benefits. One possible solution for this issue is to create research tools under the open source ethos, where all documentation about them (including their designs, building and operating instructions) are made freely available. Dubbed Open Science Hardware (OSH), this production method follows the established and successful principles of open source software and brings many advantages over traditional creation methods such as: economic savings (see Pearce 2020 for potential economic savings in developing open source research tools), distributed manufacturing, repairability, and higher customizability. This development method has been greatly facilitated by recent technological developments in fast prototyping tools, Internet infrastructure, documentation platforms and lower costs of electronic off-the-shelf components. Taken together these benefits have the potential to make research more inclusive, equitable, distributed and most importantly, more reliable and reproducible, as - 1) researchers can know their tools inner workings in minute detail - 2) they can calibrate their tools before every experiment and having them running in optimal condition everytime - 3) given their lower price point, a)students can be trained/taught with hands on classes, b) several copies of the same instrument can be built leading to a parallelization of data collection and the creation of more robust datasets. - 4) Labs across the world can share the exact same type of instruments and create collaborative projects with standardized data collection and sharing.

A Manifesto for Big Team Science

Progress in psychology has been frustrated by challenges concerning replicability, generalizability, strategy selection, inferential reproducibility, and computational reproducibility. Although often discussed separately, I argue that these five challenges share a common cause: insufficient investment of resources into the typical psychology study. I further suggest that big team science can help address these challenges by allowing researchers to pool their resources to efficiently and drastically increase the amount of resources available for a single study. However, the current incentives, infrastructure, and institutions in academic science have all developed under the assumption that science is conducted by solo Principal Investigators and their dependent trainees. These barriers must be overcome if big team science is to be sustainable. Big team science likely also carries unique risks, such as the potential for big team science institutions to monopolize power, become overly conservative, make mistakes at a grand scale, or fail entirely due to mismanagement and a lack of financial sustainability. I illustrate the promise, barriers, and risks of big team science with the experiences of the Psychological Science Accelerator, a global research network of over 1400 members from 70+ countries.

Reproducible EEG from raw data to publication figures

In this talk I will present recent developments in data sharing, organization, and analyses that allow to build fully reproducible workflows. First, I will present the Brain Imaging Data structure and discuss how this allows to build workflows, showing some new tools to read/import/create studies from EEG data structured that way. Second, I will present several newly developed tools for reproducible pre-processing and statistical analyses. Although it does take some extra effort, I will argue that it largely feasible to make most EEG data analysis fully reproducible.

Panel discussion: Practical advice for reproducibility in neuroscience

This virtual, interactive panel on reproducibility in neuroscience will focus on practical advice that researchers at all career stages could implement to improve the reproducibility of their work, from power analyses and pre-registering reports to selecting statistical tests and data sharing. The event will comprise introductions of our speakers and how they came to be advocates for reproducibility in science, followed by a 25-minute discussion on reproducibility, including practical advice for researchers on how to improve their data collection, analysis, and reporting, and then 25 minutes of audience Q&A. In total, the event will last one hour and 15 minutes. Afterwards, some of the speakers will join us for an informal chat and Q&A reserved only for students/postdocs.