Topic: multimodal masked modeling

The goal of behavioral neuroscience is to understand the relationship between neural activity and behavior. One traditional approach for studying this relationship has been to build models that either translate from the neural activity to behavior (decoding) or from behavior to neural activity (encoding). Recent advances in self-supervised learning now allow for modeling both modalities jointly. In this work, we introduce a multimodal masked modeling approach that masks portions of both behavior and neural activity, using the unmasked data to make predictions across both modalities. We evaluat