59 min
Seminar, Hosted by Kempner Institute for the Study of Natural and Artificial Intelligence at Harvard University
Back to the Future – Data Efficient Language Modeling
Computer Science, Stanford University
Recording
Abstract
Compute scaling has dominated discussion of modern language models, producing an array of algorithms that optimize performance for a given training, and sometimes inference, compute budget. As compute becomes cheaper and more abundant, however, data is becoming a bottleneck, and the ability to trade compute for data efficiency may be crucial for future model scaling. This talk examines synthetic-data and algorithmic approaches to data efficiency and shows that, in both cases, classical statistical perspectives based on nonparametric modeling and ensembling bring new insight and significant empirical benefits to modern questions of scaling and data efficiency.