Credit-based self-organization yields cortex-like topography in deep convolutional networks

Across the primate neocortex, neurons dedicated to similar functions are likely to be found physically nearby. In
the high-level visual cortex, this principle gives rise to cortical patches with distinct category selectivity that
were previously observed across many species including macaque monkeys and humans. Models of visual
cortex based on artificial neural networks (ANN) have been shown to contain internal representations that are
remarkably similar to those observed in the visual cortex. However, unit selectivity in these models emerges
without any particular spatial order (i.e. no topography). Recent work has made progress in building models
with inherent topography, however, the efficacy of these approaches has only been demonstrated in shallow
ANNs or partly-topographic ANNs [1,2]. Inspired by Kohonen’s self-organizing feature maps (SOM), we
propose a new algorithm for learning topographically organized representations in neural networks which we
call credit-based SOM (CB-SOM). Unlike Kohonen’s SOM which relies on unit activation for competitive
selection of units during learning, CB-SOM guides the selection process by considering each unit’s assigned
credit in lowering a behaviorally relevant objective.
We trained several variations of the ResNet18 architecture on Imagenet dataset while enforcing topographical
organization between units in all layers of this network according to: a) Kohonen’s SOM; b) Kohonen’s SOM
with random unit selection; c) Kohonen’s SOM with credit-based unit selection. We show that 1) convolutional
networks with topographic representations across all layers can be trained with moderate reduction in their
behavioral performance (~17-27% top-1 accuracy on Imagenet-Fig.3); 2) topographically organized categoryselective
patches emerge in neural networks trained with different variations of SOM algorithm (Fig. 1,2,4); 3)
the emerging category-selective patches in CB-SOM are substantially more brain-like compared to alternative
models (Fig.3). Together, these results highlight the potential of credit-based competitive algorithms such as
that presented here in replicating the cortical topography in modern ANNs.