Neuroformer: A Transformer Framework for Multimodal Neural Data Analysis

Systems neuroscience experiments generate multimodal data streams, including neuronal activity, stimuli, movement, and behavior events. In analysis, it can be beneficial to avoid hypothesis-driven models and instead take a data-driven approach. For this, we need a general purpose framework to handle the diverse data, and identify relationships across space, time, and modalities in a computationally efficient manner. In this work, we introduce such a framework. Our model, called the Neuroformer, can simulate neuronal populations, and jointly process an arbitrary number of additional modalities (stimuli, movement, etc.). To validate its performance, we analyzed a simulated neuronal circuit and showed that the inferred network was accurate in structure and directionality of connections. To demonstrate its utility, we used the Neuroformer to analyze large scale, multimodal data from a 2-photon calcium imaging experiment and inferred relationships between visual stimuli and neuronal activity. The Neuroformer is a flexible data-driven tool for modeling and analysis in systems neuroscience. We provide an open-source codebase for replicating and extending this work.