Locally coupled oscillatory recurrent networks learn traveling waves and topographic organization

Complex spatio-temporal neural population dynamics such as traveling waves are known to exist across multiple brain regions (Lubenov & Siapas, 2009; Muller et al., 2014), and have been hypothesized to play diverse roles from information transfer (Sato et al., 2012) to long-term memory consolidation (Muller et al., 2016). To-date, however, the empirical validation of these computational hypotheses has been hindered by the lack of a flexible and efficiently trainable model of such behavior. In this work, we introduce the Locally Coupled Oscillatory Recurrent Neural Network (LocoRNN), and show that it indeed learns to leverage traveling waves, and other well known coordinated dynamics of coupled oscillators (Kuramoto, 1975), in the service of structured sequence modeling. However, unlike previous models of such dynamics, we show that our model remains a flexible, trainable, sequence model competitive with state of the art on benchmarks such as the Hamiltonian Dynamics Forecasting Suite (Botev et al., 2021). Furthermore, when trained to model simple image sequences such as simulated retinal waves, we see that the orientation selectivity of hidden neurons becomes topographically organized, while such organization is absent when trained on unstructured noise. The resulting organization is reminiscent of orientation columns observed in the visual cortex and in line with prior work on activity-dependent organization in the visual system during development (Ackman et al, 2012). Due to local connectivity, our model is both more biologically plausible and parameter efficient than its globally coupled counterpart, the coRNN (Rusch & Mishra, 2021), while also being substantially more amenable to gradient-based training than recent spiking neural network counterparts (Davis et al, 2021) due to provably bounded gradients. Overall, we believe our results highlight the value of the LocoRNN as a novel tool for investigating the diversity of hypothesized roles of synchronous neural dynamics and their impact on computation.