1 h 9 min
Seminar, Hosted by Perimeter Institute Recorded Seminar Archive
Big Boxes, Not Black Boxes: What we can compute about LLMs, and what it may say about AGI
Hebrew University of Jerusalem
Recording
Abstract
Deep networks display recurring phenomena such as grokking, scaling behaviour, generalization despite overparameterization and failures outside training distributions. This talk surveys controlled settings where those behaviours can be explained, suggesting that language models may combine many tractable, largely interpolative mechanisms. It then distinguishes that scientific account from a more speculative argument about the role of industrial computation and human labour, and what these imply for claims of broadly extrapolative intelligence.
Topics
- deep networks
- grokking
- scaling behaviour, broader topic: behaviour
- generalization
- overparameterization, search World Wide
- training distributions, search World Wide
- language models
- interpolative mechanisms, search World Wide
- industrial computation, search World Wide
- extrapolative intelligence, broader topic: intelligence