Identifying situations in which exploration could be beneficial, and learning from such choices, is critical for survival in complex environments. Nonetheless, it is still unclear how animals adapt their choice strategies depending on the potential benefits of exploration and what the neural underpinning of this ability is. We thus adapted a 2-option forced-choice task in which 3 rhesus macaques made choices while the information and the feedback they received was being manipulated: (i) feedback was given about either just the chosen option or both the chosen and unchosen counterfactual options and (ii) feedback could or could not be used to guide future choices. We showed that monkeys chose the least rewarded option more when exploring was beneficial. We also showed that monkeys relied on not just obtained reward but also counterfactual outcomes from the unchosen option to guide their future choices. Using fMRI, we showed that chosen expected reward signals in mid cingulate cortex (MCC) and dorsolateral prefrontal cortex (dlPFC) were modulated by the availability of counterfactual outcomes at the time of choice. Specifically, we showed that when exploration was sensible (because they would only get feedback for the chosen option), MCC and dlPFC activities were negatively modulated with increased chosen expected value. In contrast, when there was no need to explore (complete feedback), MCC and dlPFC were more active with increased chosen expected value. This suggests that the suppressive effect of default actions (choosing the option with higher expected reward) might be specific to environments in which such actions trade-off with the benefits of exploration. We also saw prediction error signals for both obtained and counterfactual outcomes across orbitofrontal cortex (OFC), consistent with the idea of it as a hub for not just experienced but also imagined/ counterfactual values for state value-estimation or credit assignment.