ePoster, COSYNE 2023
Dynamics-neutral growth of stochastic stabilized supralinear networks
Puria Radmard, Wayne WM Soo, Máté Lengyel and Mate Lengyel
University of Cambridge; Department of Engineering; University of Cambridge
- Montreal, Canada
- Neuroscience, Computational Neuroscience and
Abstract
Recurrent neural networks have been trained to perform a wide variety of experimental tasks in order to understand the underlying dynamics employed by the brain to solve those tasks. However, some of the most commonly trained network architectures have not been biologically realistic, typically limiting their analogy with the brain to the level of abstract dynamical motifs. For example, they had noiseless dynamics or no separation of excitatory and inhibitory cells. The stochastic stabilized supralinear network (SSN) incorporates these features, as well as a rectified-superlinear activation function representative of cortical neurons (instead of common choices such as tanh or ReLU). Yet, the same features that make the SSN biologically realistic also make it challenging to train, as it is prone to runaway excitation through the reverberation of expansive non-linear activations. Here, we present a method to train SSNs that avoids dynamical instabilities during training. We start by randomly initializing a smaller network, and then alternate between optimizing the network and growing it by adding more neurons to it. Specifically, we add neurons in a (theoretically-motivated) dynamics-neutral way that preserves network stability. We demonstrate the effectiveness of this approach by training SSNs on: (1) the standard MNIST classification task; (2) amortized sampling-based probabilistic inference under a Gaussian scale mixture-model with a rich set of V1-like receptive fields that can sufficiently reconstruct CIFAR-10 natural images (an unachievable target for previous methods); and (3) a working memory-guided saccade task with eight different stimuli. In all examples, we find that it is nearly impossible to randomly initialize stable networks at their full sizes with similar activity levels as our trained networks, thereby highlighting the necessity of our approach. These results pave the way to training SSNs for other tasks and to understanding the circuit mechanisms of neural computations at a single cell resolution.
Topics
- Recurrent neural networks, search World Wide
- stochastic stabilized supralinear network, search World Wide
- excitatory and inhibitory cells, search World Wide
- rectified-superlinear activation function, search World Wide
- runaway excitation, search World Wide
- dynamical instabilities, search World Wide
- network stability, search World Wide
- amortized sampling-based probabilistic inference, search World Wide
- working memory-guided saccade task, search World Wide
- V1-like receptive fields, search World Wide
Details
- Session
- Poster Session I