A novel deep neural network models two streams of visual processing from retina to cortex

Human vision uses two neural pathways, namely the ventral and dorsal streams. The two streams are structurally segregated from the eyes to the primary visual cortex and continue onto the ventral temporal cortex and the dorsal parietal cortex. The ventral stream supports visual recognition and the dorsal stream guides visual attention and eye movement. Although this two-stream organization is well supported by empirical evidence, there are currently no image-computable models substantiating neural computation along two segregated pathways similar to the ventral and dorsal streams in the human brain. To fill this gap, we have implemented a two-stream neural-network model with novel features inspired by the human brain and eyes. The model learns where to look through one stream (i.e., the “where” stream) and learns what is seen through the other (i.e., the “what” stream). The two streams sample visual input with different acuity, density, and coverage similar to visual sampling by magnocellular and parvocellular retinal ganglion cells. We demonstrate that this two-stream model can attend to salient objects, perform object recognition, and explain cortical responses in humans freely watching a 115-min movie. The “what” and “where” streams in the model can selectively predict the cortical responses in the ventral and dorsal streams, respectively. The model generates human-like visual attention and recognition and its internal computational process explains the structural and functional segregation of the ventral and dorsal streams in the human brain. Whereas existing neural-network models focus on visual recognition and encoding along the ventral stream, our model accounts for both the ventral and dorsal streams and connects retinal sampling to cortical processing as integral parts of the same system. This model takes computer vision one step closer to human vision and offers novel computational insights to visual processing in humans.