Strongly interconnected neuronal populations are thought to be a substrate for memory in the brain. Dynamic connectivity relies on plasticity rules to adjust synapses, enabling learning and neural circuit formation. Networks with only excitatory synaptic plasticity can become unstable due to runaway excitation, compromising a circuit's ability to store memories. To counter this, homeostatic synaptic plasticity of inhibitory connections provide a matched negative feedback that stabilizes neural activity and any associated learning. In this study, we outline the conditions under which recurrent Excitatory (E) and Inhibitory (I) circuits with E to E and I to E plasticity produce stable dynamics. In a firing rate (FR) model, E and I plasticity produce a line attractor on which the firing rates remain fixed and learning is stable. However, in spiking networks with stochastic dynamics, these stable dynamics do not occur - we rather find a drift of synaptic weights that reflects a failure of homeostasis. If we include fluctuations in our reduced firing rate model we understand this drift as being induced by the multiplicative nature of pre- and post-synaptic neuron activity in plasticity mechanics. Such plasticity provides an product of noise variables that creates an `effective' noise-induced drift in synaptic weight evolution. This theory shows how drift can be mitigated by shared correlations among the E and I populations, effectively reducing correlated activity between synaptic fluctuations. Our work illustrates how fluctuations in the evolution of synaptic weights can lead to unexpected behavior during learning in recurrent networks.