Topic: generalization error

To learn a new task and predict accurately on new stimuli, the brain must extract patterns from limited observations. The learning rule and initial synaptic weights determine how well a network will generalize on new data. Here, we investigate how initial synaptic variability and representation learning affect the learning curves of neural networks (generalization error vs sample size). We show that two scaling regimes hold for many plausible learning rules including feedback alignment (FA), direct feedback alignment (DFA) and Hebb rules, as well as implausible gradient descent (GD). In the sm

Cortical neurons exhibit a high degree of trial-to-trial variability in response to repeated presentations of the same stimulus. We propose that this variability serves a useful computational purpose: allowing the brain to generalize from a small number of examples. To formalize this theory, we consider a Gaussian model of few-shot classification, wherein we observe a few samples of each class, and each observation has trial-to-trial variability. Our model leads to three predictions about the optimal neural response variability that minimizes generalization error: (i) the optimal variability m