The cortical dictionary: high-capacity memory in sparsely connected networks with columnar organization

Neurons with recurrent connectivity can store memory patterns as attractor states in their dynamics, forming a plausible basis for associative memory in the brain. Classical theoretical results concerning fully connected recurrent neural networks (RNNs) with binary neurons and Hebbian learning rules find that they can store at most O(N) memories, where N is the number of neurons. However, under the physiological constraint that neurons are sparsely connected, this capacity is dramatically reduced to O(K), where K is the average number of presynaptic neurons (estimated to be O(10^3 ∼ 10^4) in mammalian neocortex; note that each connection may consist of multiple synaptic contacts). This reduced capacity is inconsistent with the high capacity of human memory, particularly in language vocabularies and vision. In this work, we propose and analyze a biologically plausible mechanism that restores high-capacity memory storage under sparse connectivity. We assume that neurons in the network are organized into “columns”: in each memory, neurons from the same column encode the same feature(s), similar to columns in primary sensory areas. We propose special column-synchronizing dynamics that can utilize the columnar-coding redundancy to overcome sparse connectivity. We analytically computed the memory capacity of the model using the path-integral formulation from statistical physics. The results show that for a fixed column size M, memory capacity grows linearly with network size N until it saturates at O(MK). For optimal choice of M for each N , the memory capacity column size is O(sqrt(NK)). To our best knowledge, our results provide the first computational basis for storing a large number of memories under sparse connectivity. They also demonstrate a potential benefit of having a redundant neural code in long-term memory where many neurons encode the same memorized feature(s).