How Symmetry and Self-Coupling Shape Dynamics and Trainability of Recurrent Neural Networks

The connectivity between neurons plays a major role in the dynamics and function of all neural systems. Here, we investigate the joint effects of two previously reported connectivity motifs, partial symmetry and self-coupling, on the dynamics of recurrent neural networks (RNNs). Both motifs can be thought of as statistical deviations from i.i.d. connectivity. In particular, we define self-coupling as the values of diagonal entries and partial symmetry as the correlations across the diagonals of coupling matrices. We calculate the full Lyapunov spectrum to characterize the stability of perturbations in multiple directions of phase space. We identify two regimes in which symmetry and self-coupling affect dynamics. In the first regime, there is a small variance in the synaptic currents from recurrent input. In this regime, increasing the strength of symmetry or self-coupling results in greater dimensionality of the network dynamics. In the second regime, there is a large variance of synaptic currents, and increasing motif strength results in lower dimensionality and decreased chaos. To assess the functional implications of the effects of partial symmetry and self-coupling on dimensionality, we initialize RNNs with these motifs and train them on a multidimensional integration task. RNNs initialized with a wide range of motif strengths have variable performance. However, nonchaotic initialization as defined by an initial maximum Lyapunov exponent less than zero is the main predictor of training success across all values of initial motif strength. Our results complement previous work by considering the full Lyapunov spectrum and describing the combined effects on dynamics and trainability of different types of connectivity motifs present together. As statistical connectivity data from neural systems becomes increasingly available, we anticipate that our work will be relevant to developing new tools to find the functional and dynamical implications of these network descriptions.