Pruning for efficiency in Hopfield networks

The mammalian brain forms intricate connectivity patterns, yet its connectivity is ubiquitously sparse, enabling efficient information processing. Hopfield networks have been proposed as a schematic model of auto-associative memory retrieval in the brain; they learn to store memories by modifying their connections’ weights, proxies for biological synaptic strengths. But Hopfield networks are fully connected, which is at odds with the brain’s sparse connectivity. In this work, we ask how a Hopfield network’s memory retrieval accuracy changes when some of its connections are pruned. We hypothesize that the network’s most unstable weights are uninformative for the retrieval task. To test this, we measured the retrieval accuracy of a 32-neuron Hopfield network trained to classify white noise images, before and after we pruned its most variable weights. We found that the network maintains maximal accuracy up to a critical sparsity level, above which accuracy suddenly drops. We also found that the relationship between sparsity level and accuracy depends on the network's storage capacity known as “loading ratio”- the number of images to the number of neurons in the network. When the ratio is low, the accuracy is maximal up to a critical sparsity, then drops. Surprisingly, in the challenging regimes, the relationships are concave, rising to an intermediate sparsity level then dropping again, rather than linearly decreasing when pruning increases. As a control we found that uninformed stochastic pruning produced much poorer accuracy/sparsity tradeoffs than variance-guided pruning, demonstrating that the network can exploit its weights’ variance to achieve better energy-efficiency/accuracy tradeoff.