The dopaminergic system facilitates associative learning and motivation by signaling reward expectation and positive or negative valence in selected brain regions. Previous studies measuring somatic and axonal activity suggest a heterogeneous representation of reinforcement signaling by dopaminergic neurons, but it remains unclear how dopamine signals for expectation and valence interact in various brain regions. Here, we used an improved fluorescent dopamine sensor to record dopamine signals associated with reward predictions and valence in eight distinct striatal and non-striatal regions. We found homogeneous signaling of unexpected rewards and reward-predicting cues across the dopaminergic system. In contrast, reward omission following reward-predicting cues and aversive stimuli both produced heterogeneous signaling of dopamine. By using dimensionality reduction to examine the main axis of covariance of dopamine release across the brain, we found that reward prediction error signals are encoded by similar manifolds, whereas aversive stimulus signals are encoded by orthogonal manifolds. This suggests a high level of covariance during reward prediction error signaling that is not conserved during aversive stimulus responses. Together, our results provide evidence of contrasting reinforcement signals across the primary targets of dopaminergic pathways through which dopamine release supports learning and motivation.