Augmented Gaussian process variational autoencoders for multi-modal experimental data

Characterizing the relationship between neural population activity and behavioral data is a central goal of neuroscience. While latent variable models (LVMs) are successful in describing high dimensional data, they are typically only designed for a single type of data, making it difficult to identify structure shared across different experimental data modalities. Here, we address this shortcoming by proposing an LVM which extracts shared and independent latents for distinct, simultaneously recorded experimental modalities. We do this by combining Gaussian Process Factor Analysis (GPFA), an interpretable LVM for neural spiking data with temporally smooth latent space, with Gaussian Process Variational Autoencoders (GPVAEs), which similarly use a GP prior to characterize correlations in a latent space, but admit rich expressivity due to a deep neural network mapping to observations. We achieve interpretability in our model by constraining the latent space to partition variability into components either shared between or independent to each modality.
To make inference in our model scalable, we parameterize the latents in the Fourier domain. Using this approach, we show improved performance compared to standard inference. We validate our model on simulated multi-modal data consisting of neural spikes and smoothly rotating MNIST images. We show that the multi-modal GPVAE is able to not only identify the shared and independent latent structure across modalities accurately, but provides good reconstructions of both images and neural rates on held-out trials. Lastly, we demonstrate our framework on fly whole-brain calcium imaging alongside tracked limb positions, analyzing shared and independent subspaces between brain and behavior. We find that the subspace shared across modalities best isolates distinct fly behaviors.