Dopamine neurons reveal an efficient code for a multidimensional, distributional map of the future

Causal inference based on temporal relationships is fundamental for adaptive, intelligent behavior. However, standard value-based reinforcement learning (RL) models learn estimates of temporally discounted average future reward, leading to ambiguity about future reward timing and magnitude. Extending these algorithms to learn a set of values that differ in their sensitivity to reward prediction errors (RPEs) and temporal discounting can support inference. Midbrain dopamine (DA) neurons, known to encode RPEs, display signatures of learning about distributions of reward magnitude. However, it is unclear whether they also display signatures of learning distributions of reward timing, how such encoding might be multiplexed with reward magnitude information, and whether the joint encoding of these two dimensions allows for distributional readout of reward over time at the start of an episode. Here, we generalize distributional RL learning rules to the time domain by proposing a population coding model that optimally represents such information under resource constraints, adapting to the temporal statistics of experienced rewards. If the brain implements such a code, neural RPEs should 1) express variable sensitivity to future reward timing, 2) carry future reward distribution information and, 3) adapt to changes in the temporal statistics of reward. We tested these predictions by recording responses of optogenetically identified midbrain DA neurons to stimuli that predict rewards at variable delays and amounts. Temporal discounts varied among DA neurons, providing future reward timing information. When removing the longest delay, DA RPE responses adapted, increasing encoding accuracy on shorter reward time scales. Finally, we show that a 2D map of future reward amount over time can be decoded from the DA neuron population. Our work suggests that midbrain DA neurons reflect distributional time, in addition to distributional value RL, forming a potential substrate for fundamental computations such as causal inference, context dependent risk attitudes and impulsivity.