Operative Dimensions in High-Dimensional Connectivity of Recurrent Neural Networks

Recurrent Neural Networks (RNN) are useful models to study neural computation. Several approaches are available to train RNNs on neuroscience-related tasks and reproduce neural population dynamics. However, a comprehensive understanding of how the generated dynamics emerge from the underlying connectivity is largely lacking. Previous work derived such an understanding for specific types of constrained RNNs (Mastrogiuseppe and Ostojic, 2020; Curto et al., 2019; Biswas and Fitzgerald, 2020), but if and how the resulting insights apply more generally is unclear. In this work, we study how network dynamics are related to network connectivity in RNNs trained without specific constraints to solve two previously proposed tasks, contextual evidence integration and sine wave generation. We find that the weight matrix of these trained RNNs is consistently high-dimensional, even though the dynamics they produce is low-dimensional. Unlike in RNNs constrained to have low-rank connectivity, the functional importance of a particular dimension of the weight matrix is not predicted by the amount of variance it explains across the matrix. Despite this apparent high-dimensional structure, we show that a low-dimensional, functionally relevant subspace of the weight matrix can be found through the identification of local “operative” dimensions, which we define as dimensions in the row or column space of the weight matrix whose removal has a large influence on local RNN dynamics. Notably, a weight matrix built from only a few operative dimensions is sufficient for the RNN to operate with the original performance, implying that much of the high-dimensional structure of the trained connectivity is functionally irrelevant. The existence of a low-dimensional, operative subspace in the weight matrix simplifies the challenge of linking connectivity to network dynamics, and suggests that independent network functions may be placed in separate subspaces of the weight matrix to avoid catastrophic forgetting in continual and multitask learning.