Recent work (Aitchison et al. 2021) suggests that synaptic plasticity performs Bayesian inference and that, moreover, variability in synaptic efficacy might reflect uncertainty in the synaptic weight. Here, we consider whether similar phenomena emerge merely from minimizing synapse energetic costs during learning. Following established biophysical principles, we assume that noisy synaptic transmission is energetically cheaper than precise transmission. However, stochastic synapses may be detrimental to performance because noisy transmission adds randomness to a neural network’s activity, and so may push the network’s input-output function away from its optimum. We explored this dilemma using a combination of analysis of simple models and experiments in artificial neural networks. We found that optimizing a performance-energy trade-off results in low noise for “important” synapses, those for which reliable transmission is crucial to task performance; in contrast greater noise is afforded for less critical synapses. Interestingly, the resulting synaptic noise reflects parameter uncertainty. This implies that energy-efficient synapses should approximate Bayesian inference; we make this connection explicit. In addition, Aitchison et al. (2021) proposed that the learning rate should depend on the Bayesian posterior uncertainty, with more uncertainty leading to faster updates in light of new information. Remarkably, exactly the same phenomenon can be obtained without consideration of Bayesian inference, simply by maximizing performance under assumptions about how quickly the task changes. Jointly maximizing performance and energy efficiency leads to adaptive learning rates and synaptic noise levels, that are dependent on inferred uncertainty.