Animals naturally display a wide range of behaviours in varying contexts. Not only do their actions depend on their internal states, but also how their early sensory cortex processes information. Here, we propose that contextual top-down afferents adapt the receptive fields of sensory neurons to accommodate varying task demands. Although this view is widely supported by connectomic and electrophysiological evidence, a mechanistic and, in particular, a normative understanding is still lacking. We investigate how task-dependent top-down signals can reshape the functional mapping of sensory processing networks with fixed feedforward synaptic weights in order to solve multiple different tasks. In computational terms, the contribution of these modulations can be framed as a shift in gains and/or biases of artificial neurons. In biological neurons, we show that N-methyl-D-aspartate (NMDA) spikes in dendritic subunits are well suited to implement such modulations. Our work demonstrates that a single set of feedforward synaptic weights together with task-specific biases and/or gains can indeed solve multiple tasks, when fine-tuned by supervised learning. This approach is also suitable for transfer learning, as the top-down modulation can be adapted for new tasks without further changes in the feedforward network connections. This type transfer learning framework provides a novel criterion to evaluate the quality of a set of feedforward weights. With this in mind, we infer an unsupervised learning algorithm derived from a geometrical argument based on the structure of decision boundaries to derive synaptic weights that perform well in concert with contextual modulation. We demonstrate that the resulting method outperforms networks using common neuronal unsupervised learning algorithms. Overall, our work represents a new framework for understanding sensory processing, and sheds light on the computational mechanisms by which top-down afferents can flexibly adapt feedforward pathways for a variety tasks.