ePoster, COSYNE 2023
Variance-limited scaling laws for plausible learning rules
Alexander Atanasov, Blake Bordelon and Cengiz Pehlevan
Harvard University; Physics; Harvard University
- Montreal, Canada
- Machine Learning, Computational Neuroscience and
Abstract
To learn a new task and predict accurately on new stimuli, the brain must extract patterns from limited observations. The learning rule and initial synaptic weights determine how well a network will generalize on new data. Here, we investigate how initial synaptic variability and representation learning affect the learning curves of neural networks (generalization error vs sample size). We show that two scaling regimes hold for many plausible learning rules including feedback alignment (FA), direct feedback alignment (DFA) and Hebb rules, as well as implausible gradient descent (GD). In the small sample limit, the network is well-approximated by an infinite width network, while for large samples, the error plateaus to a final value which scales inversely with network width. We utilize a bias-variance decomposition over random initializations of the weights to show that the variance due to the initial synapse weights is primarily responsible for the deviation from infinite width behavior. We show that this variance vanishes as the network width goes to infinity, and establish connections to kernel methods. Representation learning can reduce both bias and variance for some learning rules (DFA, FA, GD) but not others (Hebb). At a fixed synapse count, the optimal allocation of resources is to use averages over several separately trained networks that have the width proportional to the square root of the number of stimuli.
Topics
- generalization error, search World Wide
- sample size, search World Wide
- initial synaptic variability, search World Wide
- representation learning, search World Wide
- learning curves, search World Wide
- feedback alignment, search World Wide
- direct feedback alignment, search World Wide
- Hebb rules, search World Wide
- gradient descent, search World Wide
- kernel methods, search World Wide
Details
- Session
- Poster Session II