Midbrain dopamine (DA) neurons send teaching signals, reward prediction errors (RPEs), to all regions of the striatum [1]. However, these signals converge with regionally segregated inputs from all levels of the cortical hierarchy [2] that vary in degree of abstraction and temporal extendedness [3]. Recent work highlights how regionally diverse DA RPEs might update parallel estimates of future reward as informed by region-specific inputs[4, 5], yet the behavioral implications of such parallel processing are poorly understood. We used photometry to measure DA release across the striatum in mice performing a value-based decision task. Importantly, two choice options were baited with reward on each trial with a fixed probability in trial blocks, meaning rewards remained baited until harvested. This introduced qualitative differences between the value of choices computed by linearly weighting past choices and rewards, and their true reward probability, which grew the longer a choice was not selected, both of which influenced behavior. DA exhibited quantitative increases in timescale, but also qualitative differences in sensitivity to task structure, from dorsolateral (DLS) to ventral striatum (VS). DLS DA reflected RPEs in relation to a weighted average of past choices and rewards, whereas VS DA reflected knowledge about the growing value of unchosen options. These data indicate that dopaminergic RPEs in different striatal regions derive from, and likely act on, a hierarchy of representations for behavioral control. We hypothesize that this reflects a scenario in which hierarchically-organized behavioral policies are learned in parallel by basal ganglia circuits, which jointly influence action selection through a computationally layered architecture for the overall control of behavior.