Equilibrium Geometry and Chaotic Dynamics in Large Recurrent Neural Networks
Large recurrent networks are important models in several fields, including neuroscience, machine learning, physics, and applied mathematics. Yet their dynamics are difficult to study directly, because high-dimensional nonlinear systems can exhibit rich behavior that is hard to summarize in terms of individual trajectories. In this talk, I will discuss an approach that seeks to understand such dynamics through the structure of the network’s equilibria. I will focus on a random balanced network of threshold-linear units that undergoes a transition from a single stable equilibrium to extensive chaos as the disorder strength crosses a critical value. Using a combination of Kac–Rice theory, replica calculations, numerical root-finding, and dynamical mean-field theory, we show that the chaotic regime contains an exponentially large number of equilibria. These equilibria are all saddles, but with only a fractionally small number of unstable directions. Surprisingly, despite the completely random connectivity, the equilibria are not scattered randomly through phase space. Instead, they are strongly correlated and confined to a comparatively small region. The chaotic attractor lies within this same region, suggesting a direct geometric link between the organization of unstable equilibria and the collective structure of the dynamics. This picture helps explain why networks with extensive chaos can nevertheless display dynamics dominated by a relatively small number of collective modes. More broadly, the results suggest that the geometry of equilibria provides a useful complementary perspective to dynamical mean-field theory for understanding high-dimensional neural dynamics. Presented in the van Vreeswijk Theoretical Neuroscience Seminar series (formerly WWTNS) on 2026-05-27. Recording duration: 00:46:40.
Computation Through Neuronal-Synaptic Dynamics
Computations in neural circuits are often construed as being implemented through the coordinated dynamics of neurons. In this picture, the role of synaptic connectivity is to sculpt neuronal dynamics to implement computations of interest. Of course, synapses are not static but change on a variety of timescales, including fast timescales comparable to those of neurons. Thus, a more accurate view of computation in neural circuits may involve the coupled dynamics of neurons and synapses. This form of computation is closer to what is implemented by Transformers via an equivalence between ongoing synaptic plasticity and self-attention. I will first describe a nonlinear recurrent neural-network model with ongoing Hebbian dynamics of “fast” synapses atop unstructured “slow” synapses. I will then describe two computations implemented through neuronal-synaptic dynamics, which can be studied in this model using techniques including dynamical mean-field theory and random-matrix theory. First, there exists a novel phase termed “freezable chaos” in which a stable fixed point of neuronal dynamics is continuously destabilized by synaptic dynamics. This allows for the creation of a stable fixed point at any neuronal state visited by the network by halting synaptic plasticity. Second, I will describe an effect termed “persistent oscillations” in which, following stimulation by a periodic signal, a plastic network continues to autonomously reproduce a similar signal for a duration exceeding any intrinsic timescale in the system. Thus, ongoing Hebbian plasticity can provide a dynamic form of working memory, complementing the static form provided by freezable chaos. Ongoing experimental work suggests that this effect is realized in cortical organoids. Overall, this line of work suggests that synapses should be promoted to first-class dynamical degrees of freedom in our conceptual understanding of neural-circuit function. Presented in the van Vreeswijk Theoretical Neuroscience Seminar series (formerly WWTNS) on 2026-01-14. Recording duration: 00:43:14.
Theory of gating in recurrent neural networks
Recurrent neural networks (RNNs) are powerful dynamical models, widely used in machine learning (ML) for processing sequential data, and also in neuroscience, to understand the emergent properties of networks of real neurons. Prior theoretical work in understanding the properties of RNNs has focused on models with additive interactions. However, real neurons can have gating i.e. multiplicative interactions, and gating is also a central feature of the best performing RNNs in machine learning. Here, we develop a dynamical mean-field theory (DMFT) to study the consequences of gating in RNNs. We use random matrix theory to show how gating robustly produces marginal stability and line attractors – important mechanisms for biologically-relevant computations requiring long memory. The long-time behavior of the gated network is studied using its Lyapunov spectrum, and the DMFT is used to provide a novel analytical expression for the maximum Lyapunov exponent demonstrating its close relation to relaxation-time of the dynamics. Gating is also shown to give rise to a novel, discontinuous transition to chaos, where the proliferation of critical points (topological complexity) is decoupled from the appearance of chaotic dynamics (dynamical complexity), contrary to a seminal result for additive RNNs. Critical surfaces and regions of marginal stability in the parameter space are indicated in phase diagrams, thus providing a map for principled parameter choices for ML practitioners. Finally, we develop a field-theory for gradients that arise in training, by incorporating the adjoint sensitivity framework from control theory in the DMFT. This paves the way for the use of powerful field-theoretic techniques to study training/gradients in large RNNs.