To successfully inform behavior, cortical circuits must produce consistent patterns of neuronal activity within and across trials that allow for reliable stimulus discrimination. Under this perspective, any response variability appears detrimental to sensory systems as it severely limits stimulus discriminability. Here, we propose a beneficial role of response variability: to facilitate faster learning in feedforward-recurrent networks through a self-stabilizing feedback mechanism. We explore this hypothesis in the framework of learning novel stimulus representations in pre-structured recurrent networks that receive input from a feedforward network. In the visual system, recent work has shown a drastic change in response reliability following sensory experience that is consistent with re-aligning novel sensory inputs and a structured recurrent network (‘feedforward-recurrent alignment hypothesis’) [1]. Here, we study linear rate models, in which we adapt both recurrent network and feedforward input connectivity to optimize feedforward-recurrent alignment and compared with coactivity-dependent plasticity rules such as Hebbian learning. First, we show that such an increase in feedforward-recurrent alignment is inherently consistent with Hebbian learning. Next, exploring the effect of the input dimensionality on the learning speed, we find that initially higher-dimensional input – consistent with high initial response variability – can lead to faster convergence of the optimization process. These results suggests that variable and inconsistent responses to novel sensory input can provide a flexible starting point for experience to rapidly drive changes in the response properties, highlighting a beneficial role for initial response variability.