Topic: counting

Andrew Dai joins The Information Bottleneck to discuss language-model pre-training, next-token prediction, training-data quality and visual reasoning. The conversation examines counting and image understanding, world models and JEPA. Hosted by Ravid Shwartz-Ziv and Allen Roush.

Although animals can estimate numerical quantities, true counting and arithmetic abilities are unique to humans and are inextricably linked to symbolic competence. However, our unprecedented numerical skills are deeply rooted in our neuronal heritage as primates and vertebrates. I argue that numerical competence in humans is the result of three neural constraints. First, I propose that the neuronal mechanisms of quantity estimation are part of our evolutionary heritage and can be witnessed across primate and vertebrate phylogeny. Second, I suggest that a basic understanding of number, what num