Modeling the orbitofrontal cortex function in navigation through an RL-RNN implementation

Since its initial conceptualization by Edward Tolman, cognitive maps – i.e., task-derived internal representations of an environment – have been found in multiple brain regions. In mammalian brains, the orbitofrontal cortex (OFC) can produce a flexible cognitive map underpinning model-based learning. Recent results have shown that this function is compatible with the canonical value-encoding role ascribed to OFC, whereby this region guides decision-making by providing a representation of reward. Hence, recent theories have suggested that the OFC integrates activity from other brain regions and value information to generate a goal-oriented state-space representation that does not remap and can aid generalization.

Here, we use recurrent neural networks in a reinforcement learning (RL-RNN) paradigm to reproduce the function of OFC in a goal-directed navigational task. We assumed this region operates as a value-based network separated from action planning, as in actor-critic or REINFORCE-type systems. Our model recreated the switching behavior observed experimentally between two goals, which requires model-based learning to avoid catastrophic forgetting of the attractor states. Moreover, we reproduced the low-dimensional and single-neuron activity recorded in vivo, and show that such a system accounts for model-based learning. We also provide some intuitions into how the OFC supports a stable encoding of goal locations even under changed reward contingencies, or new goals are introduced without plasticity. Notably, our work supports the view of the OFC as both a cognitive map and a value-based system.