Understanding Auditory Cortex with Deep Neural Networks

Recent work [Yamins 2014, Bashivan 2019, Bao 2020] has shown that the initial responses of individual neurons in the ventral stream of visual cortex can be predicted with high accuracy by deep neural networks trained on challenging visual classification problems. Similar results have been obtained for voxels of auditory cortex with fMRI [Kell 2018]. But the relatively poor time resolution of fMRI precludes capturing temporal structure below about 1 Hz, which is known to be an important characteristic of auditory processing [Downer 2020]. Here we investigate the relationship between sequences of neural spiking activity, in response to audio stimuli, in the auditory cortex of squirrel monkeys; and sequences of hidden-layer representations in a deep neural network (DNN) trained to convert speech audio to text. We show: (1) Layers of the trained—but not the untrained—DNN can explain the activity of neurons in core and belt areas of auditory cortex, at temporal frequencies of 50 Hz and higher, and with noise-corrected correlations as high as 0.94. (2) Different regions of auditory cortex correlate well with different layers of the DNN model, suggesting the existence of topographic organization not based strictly on tonotopy. (3) The DNN can be used to generate “optimal” audio inputs—i.e., that maximize the predicted neural response [Bashivan 2019]—which in some cases turn out to resemble chirps, a common call of the squirrel monkey. The success of this method has important implications for understanding the computational basis of auditory cortex, since the “object-space” of non-primary auditory areas is unknown.