Topic: training data

Perfectly fitting the training data does not guarantee good generalization. For two idealized data distributions, this talk characterizes how frequently linear interpolating classifiers attain a given generalization error. Large-deviation and concentration results show why typical interpolators can perform worse than solutions reached by gradient descent and motivate the problem of sampling them.

Damian Borth describes learning from trained neural networks themselves as a new source of training data. The conversation explores representations of model weights, transferring knowledge across architectures and tasks, and the possibility of specializing models with less computation by reusing what existing networks have learned.

Artificial neural networks (ANN) have emerged as computationally plausible models of human language processing. A major criticism of these models is that the amount of training data they receive far exceeds that of humans. Here, we use two complementary approaches to ask how the models’ ability to capture human neural and behavioral responses to language is affected by the amount of training data. First, we evaluate GPT-2 models trained on 1 million, 10 million, 100 million, or 1 billion tokens against a neural (fMRI) and a behavioral (reading times) benchmark. Because children are exposed to