Dynamics-neutral growth of stochastic stabilized supralinear networks

Recurrent neural networks have been trained to perform a wide variety of experimental tasks in order to understand the underlying dynamics employed by the brain to solve those tasks. However, some of the most commonly trained network architectures have not been biologically realistic, typically limiting their analogy with the brain to the level of abstract dynamical motifs. For example, they had noiseless dynamics or no separation of excitatory and inhibitory cells. The stochastic stabilized supralinear network (SSN) incorporates these features, as well as a rectified-superlinear activation function representative of cortical neurons (instead of common choices such as tanh or ReLU). Yet, the same features that make the SSN biologically realistic also make it challenging to train, as it is prone to runaway excitation through the reverberation of expansive non-linear activations. Here, we present a method to train SSNs that avoids dynamical instabilities during training. We start by randomly initializing a smaller network, and then alternate between optimizing the network and growing it by adding more neurons to it. Specifically, we add neurons in a (theoretically-motivated) dynamics-neutral way that preserves network stability. We demonstrate the effectiveness of this approach by training SSNs on: (1) the standard MNIST classification task; (2) amortized sampling-based probabilistic inference under a Gaussian scale mixture-model with a rich set of V1-like receptive fields that can sufficiently reconstruct CIFAR-10 natural images (an unachievable target for previous methods); and (3) a working memory-guided saccade task with eight different stimuli. In all examples, we find that it is nearly impossible to randomly initialize stable networks at their full sizes with similar activity levels as our trained networks, thereby highlighting the necessity of our approach. These results pave the way to training SSNs for other tasks and to understanding the circuit mechanisms of neural computations at a single cell resolution.