Multi-object memory and prediction in the primate brain

Primates excel at reasoning about physical scenes comprised of objects, flexibly generalizing to novel compositions of objects. Many cognitive theories attribute this capacity to structured representations in working memory called “object files” that encode object features in independent subspaces that can be readily retrieved and processed. However, these theories remain largely un-tested in the primate brain. We developed a task for non-human primates (NHPs) that requires remembering the positions and identities of multiple objects on a visual display. In some conditions the objects move under deterministic physics, requiring the subject to additionally track their kinematics. NHPs were able to perform this task and generalized to a variety of held-out conditions including (i) novel object identities, (ii) novel spatial configurations, and (iii) novel kinematics. Several brain regions in the frontal cortex have been implicated in the representation and processing of visual information in working memory, including the frontal eye fields (FEF) and dorsomedial frontal cortex (DMFC). Accordingly, we recorded simultaneously from large populations of single neurons in FEF and DMFC in two NHPs while they performed the task. To test different possible neural instantiations of the object-file theory, we formulated specific neural hypotheses, developed metrics to distinguish between them, validated those metrics in artificial networks, and applied those metrics to our neural data. We found that neural activity in both FEF and DMFC during working memory is consistent with an object file representation where the assignment of objects to files is determined by the spatial configuration of the objects in the visual field. Moreover, we found that neural activity is not consistent with a variety of alternative models. Our results reject some but not all models of multi-object representation in the brain.