Robust Sequence Recall in Asymmetric Modern Hopfield Networks

Sequential memory is an essential attribute of natural and artificial intelligence that enables agents to encode, store, and retrieve complex sequences of stimuli and actions. Biologically plausible models of sequential memory have been proposed where recurrent neural networks are trained with temporal asymmetric Hebbian (TAH) rules. However, these networks have limited capacity, defined by the maximum length of sequence they can reliably recall, due to overlaps between stored memories reducing the signal to noise ratio. Inspired by recent work in Modern Hopfield Networks (MHN), we expand the capacity of this model by introducing a polynomial nonlinearity to the interaction term. This enhances the separation of patterns. We derive theoretically the improved scaling law for the maximal length of the stored sequence of patterns and verify this result with numerical simulation. Furthermore, we show that our extended model can be implemented as a biologically plausible network with similarities to the cortico-basal ganglia-thalamo-cortical loop, responsible for selecting and generating complex motor sequences.