Juvenile songbirds learn to imitate adult vocalizations. Song acquisition and production is governed by a dedicated neural circuitry that involves two parallel pathways: a cortical pathway required for production and a basal ganglia-thalamo-cortical (BG) pathway necessary for plasticity. The BG pathway induces variability in production during vocal exploration, receives a performance signal via midbrain dopaminergic projections and drives a motor bias that corrects vocal errors. This dopamine-modulated change in vocal output, induced by the BG is gradually consolidated within the cortical pathway. Reinforcement learning (RL) has been widely hypothesized to underlie sensorimotor learning, including song learning. However, pure RL approaches may result in non-optimal solutions under uneven reward contours in a continuous action space. We propose to re-interpret the role of the dual pathway architecture in songbirds and to help overcome these limitations. We posit that the BG pathway conducts daily exploration by inducing large jumps in the vocal exploration. BG-driven vocal exploration is modulated over both the daily and weekly timescales, as the cortical pathway matures. The cortical pathway gradually consolidates BG reward-modulated exploration with Hebbian learning. We demonstrate how this process essentially implements simulated annealing. In an artificial reward landscape, the dual pathway network can reach the global optimum in a number of trials, comparable with the learning period of songbirds (60 days of learning, 500 trials per day). We further emulate an experimental protocol to induce plasticity in adult birds by locally modulating the reward profile around the global optimum after convergence. As observed experimentally, our model network adapts to this change in reward profile and modifies its vocal exploration profile. To further test this interpretation, we shall contrast with ongoing electrophysiological investigations in the output nuclei of these two pathways in zebra finches, subjected to a distorted feedback protocol.