An RNN model of planning explains hippocampal replay and human behavior

When interacting with complex environments, humans can rapidly adapt their behavior to changes in task or context. This flexibility has been suggested to result from fast recurrent dynamics in the prefrontal network, acquired through a process of meta-reinforcement learning (Wang et al., 2018). However, these models fail to capture an important aspect of human behavior: when adapting to new information, we spend variable and often long periods of time contemplating possible futures before acting. For such planning to be rational, its benefits to future behavior must at least compensate for the time spent thinking. Here, we capture these properties of human behavior by developing a neural network model where not only actions, but also planning itself, are controlled by prefrontal cortex. The model consists of a meta-reinforcement learning agent augmented with the ability to perform ‘simulation-based planning’ in the form of policy rollouts. Our results demonstrate that the agent learns to plan when planning is beneficial, explaining the empirical variability in human thinking times. Intriguingly, the patterns of policy rollouts employed by the artificial agent, and their effect on behavior, closely resemble patterns of rodent hippocampal replays in a spatial navigation task studied by Widloski et al. (2022). Our work suggests that the brain implements simulation-based planning through prefrontal-hippocampal interactions, where hippocampal replays are triggered by - and in turn adaptively affect - prefrontal dynamics.