Variance-limited scaling laws for plausible learning rules

To learn a new task and predict accurately on new stimuli, the brain must extract patterns from limited observations. The learning rule and initial synaptic weights determine how well a network will generalize on new data. Here, we investigate how initial synaptic variability and representation learning affect the learning curves of neural networks (generalization error vs sample size). We show that two scaling regimes hold for many plausible learning rules including feedback alignment (FA), direct feedback alignment (DFA) and Hebb rules, as well as implausible gradient descent (GD). In the small sample limit, the network is well-approximated by an infinite width network, while for large samples, the error plateaus to a final value which scales inversely with network width. We utilize a bias-variance decomposition over random initializations of the weights to show that the variance due to the initial synapse weights is primarily responsible for the deviation from infinite width behavior. We show that this variance vanishes as the network width goes to infinity, and establish connections to kernel methods. Representation learning can reduce both bias and variance for some learning rules (DFA, FA, GD) but not others (Hebb). At a fixed synapse count, the optimal allocation of resources is to use averages over several separately trained networks that have the width proportional to the square root of the number of stimuli.