Skip to contentSkip to filters

We use cookies for analytics.

Back to the Future – Data Efficient Language Modeling

Tatsunori Hashimoto

Computer Science, Stanford University

Recording

59 min

Abstract

Compute scaling has dominated discussion of modern language models, producing an array of algorithms that optimize performance for a given training, and sometimes inference, compute budget. As compute becomes cheaper and more abundant, however, data is becoming a bottleneck, and the ability to trade compute for data efficiency may be crucial for future model scaling. This talk examines synthetic-data and algorithmic approaches to data efficiency and shows that, in both cases, classical statistical perspectives based on nonparametric modeling and ensembling bring new insight and significant empirical benefits to modern questions of scaling and data efficiency.

Topics