Globally gated deep linear networks

Gating represents multiplicative interactions in neural systems, and are well known motifs of biological neural circuits. Recently neural networks with gating have attracted increased attention. Gated Linear Networks (GLNs) have been studied for their interesting capabilities such as biologically plausible learning and reduced forgetting in sequential learning. The network has also been shown to mimic some aspects of cerebellar architecture and plasticity. However, there has been virtually no theoretical understanding of the capabilities and limitations of GLNs. To close this gap, we introduce Globally Gated Deep Linear Networks (GGDLNs) where gating units are shared among all processing units in a given layer, decoupling the architectures of the nonlinear, unlearned gating and the learned linear processing motifs. This allows for rigorous theoretical analysis of these interesting systems.
We derive exact equations for performance of GGDLNs. We find that the statistics of the network outputs after learning can be expressed in terms of the input similarity matrix and the inner product of the gating units which undergoes modification through a task-dependent matrix. Interestingly, this matrix is directly related to the covariance of the readout weights, generating experimentally verifiable predictions of network connectivity given specific tasks. Using our theory, we study several aspects of the network. We show that GGDLNs with sufficient gating units behave similarly to fully trained nonlinear networks. We found that the network can achieve feature selection by adjusting the task-dependent matrix to select important gating units for the relevant tasks. Additionally, we evaluate the network’s ability for context-dependent learning by incorporating task-relevant information into the gating units, mimicking top-down contextual signals from the mossy fiber pathway. In summary, we derive exact theoretical predictions of performance and connectivity in GGDLNs trained on any specified tasks, providing insights for understanding learning with dendritic gating architectures in the brain.