Reinforcement learning at multiple timescales in biological and artificial neural networks

In order to thrive in complex environments, animals and artificial agents have to learn to act adaptively to maximize fitness and rewards. Reinforcement learning (RL) algorithms have been successful at training artificial agents to superhuman performance in an array of complex tasks and the activity of dopaminergic neurons has been described by one of their key components, the reward prediction error (RPE). However, classical RL occurs at a single timescale, set by the discount factor, while agents have to make decisions at multiple timescales and behaviorally, humans and animals are known to discount non-exponentially in many situations. Here, we show that agents learning at multiple timescales possesses distinct computational benefits, including (1) a richer representation of the temporal evolution of rewards, (2) an ability to infer reward timing before learning converges and (3) better adaptation to alternative discount mechanisms to produce more accurate estimates and behave more efficiently. Next, by recording optogenetically identified single dopaminergic neurons in mice performing two behavioral tasks, a cued delayed reward task and navigation in a 1-D virtual reality track, we show that they encode RPEs at a diversity of timescales through a diversity of discount factors. This diversity allows us to (1) decode reward timing using a parameter-free approach (2) explain heterogeneous ramping activity across dopaminergic neurons while approaching reward by single neurons with different discount factors experiencing a common value function (or reward expectation) and (3) show that the discount factor of individual neurons is correlated across the two tasks, suggesting that it is a cell-specific property and setting constraints on the biological implementations of multi-timescale RL. Together with other advances such as distributional RL in the reward domain, our work opens new directions for efficient RL algorithms based on distributional coding along several dimensions in both artificial and biological agents.