ePoster, COSYNE 2023
Emergent compositional reasoning from recurrent neural dynamics
William Tong and Cengiz Pehlevan
Harvard University; School of Engineering and Applied Sciences; Harvard University
- Montreal, Canada
- Computational Neuroscience, Cognition and
Abstract
The ability to reason compositionally is a hallmark of intelligent behavior. This algebraic capacity to assemble new concepts from novel combinations of existing components lies at the heart of many human intellectual endeavors: language, math, science, philosophy, and much more. Further, recent discoveries in animal cognition indicate that compositional reasoning may not be a uniquely human trait --examples include numeracy in non-human primates, spatial navigation using cognitive maps in rodents, and even the use of basic syntax in songbirds (Tervo, Tenenbaum, & Gershman 2016) suggesting a widespread and foundational role in natural intelligence. Despite the importance of this ability, it remains unclear how compositional reasoning manifests in the brain. As a first step towards understanding the relevant neural dynamics, we present an analysis on the emergence of compositional reasoning in recurrent neural networks (RNN) trained on a simple binary addition task. We discover that 1) task-optimized RNNs consistently develop an attractor landscape whose geometry encodes a partially-compositional representation of the task, and 2) RNNs that are expressive enough to learn a fully-compositional solution nonetheless fail to do so through gradient-based methods alone. Rather, models may require a global search strategy like an evolutionary algorithm to identify optimal basins in the loss landscape that converge towards fully-compositional solutions. Together, these findings offer an initial account of how collective neural activity can implement an inference procedure exhibiting basic compositionality.
Topics
- compositional reasoning, search World Wide
- recurrent neural networks (RNN), search World Wide
- binary addition task, search World Wide
- attractor landscape, search World Wide
- partially-compositional representation, search World Wide
- fully-compositional solution, search World Wide
- gradient-based methods, search World Wide
- evolutionary algorithm, search World Wide
- loss landscape, search World Wide
- collective neural activity, search World Wide
Details
- Session
- Poster Session II