Prioritizing experience replay when future goals are unknown

Although hippocampal place cells prominently "replay" nonlocal trajectories, the computational function of these events remains elusive. One hypothesis holds that replay plans routes to current goals by propagating information about rewards over space. An alternative view contends that replay learns and maintains routes more generally, separate from current goals or plans, to update a "cognitive map". The former view has been formalized in a reinforcement learning framework (Mattar and Daw, 2018), which predicts which trajectories should be replayed preferentially ("prioritized") to support planning. However, recent data appear to contradict this perspective, showing a disconnect between replayed destinations and current goals (Gillespie et al., 2021), and no similar theory exists to formalize the alternative perspective. Here, we generalize the Mattar account to address replay for a map of routes to many locations (a variant of the successor representation), when which of these contain reward is unknown or expected to change in the future. This leads to a prioritization strategy that evaluates candidate trajectories for replay according to a learned distribution over possible future goal locations, rather than with respect to the current reward function. We show that replay in an agent with this objective explains experimental results in which replay is systematically focused on different goals than the animal's current behavior.