Date / Location: 18 March / II-003

Predicting the future outcomes of actions forms the basis for reinforcement learning to shape goal-directed behaviours. Recent work showed that learning based on predicting future sensory experience using the current state and action of an agent leads to representations that resemble those in the brain, for instance place and grid cells of the medial temporal lobe. Here we ask if combining ongoing predictive learning of sensory events and of notional value of actions leading to rewards forms representations that enable more efficient goal-directed learning than reinforcement learning alone. Simulating a simple T-maze environment, we find that once a recurrent network is trained to predict future sensory inputs based on actions, an attractor landscape forms resembling hippocampal place cells. Next, we introduce cued rewards, and train the network to predict state-action Q-values which are used to guide subsequent behaviour. A network previously exposed to the same environment without rewards learns the task faster than a network trained using Q-learning alone, or without previous exposure. Interestingly, this training paradigm causes non-local neural activity to sweep forward in space at decision points, anticipating the future path to a rewarded location. Moreover, prevalent choice and cue-selective neurons form in this network, again recapitulating experimental findings. Together, these results indicate that a simple combination of predictive, unsupervised learning of environment structure and of reinforcers yields efficient representations to support goal-directed behaviour and exhibit dynamics also found experimentally in the hippocampus when learning similar tasks.