Credit-based self-organization yields cortex-like topography in deep convolutional networks

Across the primate neocortex, neurons dedicated to similar functions are likely to be found physically nearby. In the high-level visual cortex, this principle gives rise to cortical patches with distinct category selectivity that were previously observed across many species including macaque monkeys and humans. Models of visual cortex based on artificial neural networks (ANN) have been shown to contain internal representations that are remarkably similar to those observed in the visual cortex. However, unit selectivity in these models emerges without any particular spatial order (i.e. no topography). Recent work has made progress in building models with inherent topography, however, the efficacy of these approaches has only been demonstrated in shallow ANNs or partly-topographic ANNs [1,2]. Inspired by Kohonen’s self-organizing feature maps (SOM), we propose a new algorithm for learning topographically organized representations in neural networks which we call credit-based SOM (CB-SOM). Unlike Kohonen’s SOM which relies on unit activation for competitive selection of units during learning, CB-SOM guides the selection process by considering each unit’s assigned credit in lowering a behaviorally relevant objective. We trained several variations of the ResNet18 architecture on Imagenet dataset while enforcing topographical organization between units in all layers of this network according to: a) Kohonen’s SOM; b) Kohonen’s SOM with random unit selection; c) Kohonen’s SOM with credit-based unit selection. We show that 1) convolutional networks with topographic representations across all layers can be trained with moderate reduction in their behavioral performance (~17-27% top-1 accuracy on Imagenet-Fig.3); 2) topographically organized categoryselective patches emerge in neural networks trained with different variations of SOM algorithm (Fig. 1,2,4); 3) the emerging category-selective patches in CB-SOM are substantially more brain-like compared to alternative models (Fig.3). Together, these results highlight the potential of credit-based competitive algorithms such as that presented here in replicating the cortical topography in modern ANNs.