Hebbian learning of a multi-layered cerebellar network with quadratic memory capacity

A fundamental goal of neuroscience is the functional understanding of neural circuit architecture. Previous studies revealed the importance of network depth in visual and auditory circuits, but it remains elusive why predominantly non-convolutional circuits, such as the cerebellum or olfactory systems, have multiple feedforwardly connected layers. Recent deep learning works revealed that a two-hidden-layer network has a quadratically more memory capacity than a single-hidden-layer network with the same number of hidden layer neurons, suggesting the importance of the network depth for memory capacity. However, those studies were based on non-local tuning of synaptic weights; thus, whether the depth provides memory gain to the brain under biologically-plausible learning remains unclear. Here, we show analytically, and demonstrate numerically, that error-driven Hebbian learning is sufficient to achieve quadratic memory capacity in a two-hidden-layer network with gating when the activity sparseness is optimized. This gated two-hidden-layer network exhibits several similarities with the cerebellum: First, the cerebellum mainly has two hidden layers, granule cells and Purkinje cells, between mossy fibers (input) and cerebellar nuclei (output). Secondly, learning takes place at the connections between granule cells and Purkinje cells, which is modulated multiplicatively by the presynaptic activity, an error signal from climbing fiber, and shunting inhibition from stellate cells. Lastly, the number of granule cells is similar to the total number of Purkinje cell dendritic branches, though much more numerous than the Purkinje cell population. Our work thus advances the Marr-Albus-Ito model of the cerebellum from a deep learning perspective.