Neural activity in the cortex exhibits a wide range of timescales, allowing animals to respond to rapidly changing sensory stimuli while maintaining information in working memory. Recent studies show how top-down inputs representing locomotive state modulates these timescales differently across visual areas in mice. In particular, the sign of correlations between speed and neural activity varies by region, highlighting the interplay between bottom-up sensory inputs, top-down pathways and connectivity motifs in shaping cortical dynamics. In addition, cortical circuits comprise a large diversity of cell types connected in non-random ways, which are selectively targeted by distinct modulatory inputs. Due to this complexity, understanding how top-down projections to precise cell types affect the timescales of cortical network dynamics remains elusive. Here, we investigate how networks with connectivity generated by experimentally measured synaptic plasticity rules respond to different forms of top-down modulation. We train large-scale spiking networks with multiple inhibitory interneurons (parvalbumin PV, somatostatin SST and vasoactive intestinal peptide VIP) using different spike-timing-dependent plasticity (STDP) rules at the different synapses. We find that, by incorporating or omitting STDP rules at SST-to-excitatory synapses, the circuits are primarily dominated either by disinhibition or inhibition of excitatory cells, resulting in different dynamical regimes. In the two networks, different forms of top-down modulation can speed up or slow down neural activity, affect the encoding speed and shift the networks between being primarily recurrent or feedforward driven. Finally, we show how the two networks can be integrated into a single biologically plausible cortical circuit. This unified network retains the individual properties of both original networks and also provides an explanation for the experimentally observed changes in correlation sign across visual areas during locomotion. This work highlights the role different plasticity mechanisms have in structuring networks to process information at a range of timescales and adapt to rapidly-changing environments.