Resources
Authors & Affiliations
Oliver Vikbladh,Evan Russek,Neil Burgess
Abstract
A variety of algorithms have been proposed to account for goal-directed planning, operationalized as sensitivity to reward revaluation i.e. flexibility in the face of reward changes. These algorithms use different representations which trade off computational cost with flexibility in the face of changes in the world. In particular, model-based (MB) reinforcement learning uses tree-search through 1-step relational representations that are often thought of as ‘semantic’. In contrast, models like the successor representation (SR) store representations of sequential experience. These algorithms can be distinguished based on MB sensitivity to transition revaluation, i.e. flexibility following changes in state-state transition relationships, which the SR lacks.
It is believed that consolidation changes the way memories are represented and structured – transferring them from hippocampus to cortex and making them more abstract and semantic. This is important in light of recent findings, showing a transient hippocampal involvement in goal-directed planning, i.e. the hippocampus is required immediately after learning but not a week later (Bradfield et al, Nat Neuro, 2020). Further, the hippocampus has been proposed to specifically support SRs (Stachenfeld et al., Nat Neuro, 2017). We therefore sought to test how consolidation impacts the representations used for planning.
To do this, we developed a new planning task which can detect either MB use of 1-step transition models, or use of sequential experience, like SRs. After training on fixed sequences, unique reward and transition revaluation problems are posed so as to map the relative use of either type of representation. We demonstrate that a week of memory consolidation pushes behaviour from SR-like insensitivity to transition revaluation towards flexible MB use of 1-step transitions that is sensitive to both reward and transition revaluations.