A role for cortical pattern separation in enhancing visual memory

Humans have a remarkable ability to remember whether they've seen an image before, with considerable visual detail. Unlike the visual perceptual confusions of images, visual memory confusions are not approximately predicted by the distances between population representations in inferotemporal cortex (ITC), or its analogs in artificial neural networks. The leading hypothesis to account for this discrepancy proposes that the hippocampus (HC) applies a nonlinear transformation of its cortical inputs to better separate them before memory storage, a mechanism called hippocampal pattern separation (HPS). Less appreciated is evidence that cortical adaptation can also separate population responses under real-world and often-encountered conditions, when visual inputs are enriched for one type (for example, seeing a dog among many other dogs at a dog park). We call this mechanism adaptation-induced cortical pattern separation (aCPS), and we propose that it enhances visual memory. To determine relative contributions of HPS and aCPS to visual memory, we recorded from monkey ITC and HC during a single-exposure visual familiarity task. Images appeared in one of two types of short blocks: random blocks with images from many categories, or categorical blocks enriched for a single object category. We found evidence for both aCPS and HPS in our data, however, aCPS was 4.5-fold more effective at separating the population responses to images than HPS. Additionally, we estimated the behavioral consequence of aCPS to be considerable: a 35% enhancement in memory performance relative to a hypothetical world without it. Finally, to evaluate the plausibility that aCPS operates via adaptation, we extended the IT-layer of an artificial neural network to simulate this mechanism and found that the resulting model recapitulated notable signatures of our data. Together, these results establish a role for cortical pattern separation in enhancing visual memory under conditions that are often encountered during real world viewing.