Fitting normative neural sampling hypothesis models to neuronal response data

A prominent theory of sensory perception advocates that perception in the brain is implemented via probabilistic inference. The neural sampling hypothesis (NSH) posits that neuronal responses to a stimulus represent samples from the posterior distribution over latent world state variables (e.g., object identity) that underlie the stimulus. Existing work on NSH commonly evaluates qualitative agreement of experimental data with simple generative models of the stimulus and does not fit NSH models to experimentally observed sensory population responses.
We propose a novel formulation for NSH that allows us to directly fit NSH models to recorded stimulus-response pairs, and to formulate more flexible generative models. We formalize NSH as an equivalence between the distribution over stimulus-conditioned responses and the posterior distribution over stimulus-conditioned latent world-state variables. This enables us to fit generative models under NSH to responses and stimuli, including existing NSH models. Furthermore, we use a normalizing flow-based neural-network model and learn the generative model directly on image-response pairs. Our formulation allows us to directly compare NSH models to existing DNN-based encoding models of stimulus-conditioned responses. We fitted a Hoyer-Hyva ̈rinen NSH model directly on macaque V1 responses to natural images, and compared its performance to a state-of-the-art deep neural-network system identification models. We found that the NSH model was outperformed by even a simple linear- nonlinear model. While this is somewhat expected, the size of the performance gap clearly indicates that current NSH models are too simple for real responses and motivates the development of more complex generative models. Overall, our work is an important first step toward a more quantitative evaluation of NSH models and provides a novel framework that will let us learn the generative model directly on data, paving the way for a better understanding of probabilistic computational principles that underlie perception and behavior.