The nanoscale arrangement of chemical synapses is thought to be critical for establishing synapse strength but whether this arrangement is sensitive to changes in in vivo activity is not known. Pyramidal neurons in the cortex receive a multitude of excitatory and inhibitory inputs that determine the generation of the action potential (AP) at the axon initial segment (AIS). Chandelier cells (ChCs), a GABAergic interneuron subtype, targets the AIS of pyramidal neurons through axo-axonic synapses and is therefore thought to strongly modulate neuronal output. We used super-resolution microscopy (Stochastic Optical Resolution Microscopy - STORM) and the expression of fibronectin intrabodies (FingRs) targeting gephyrin to visualize the postsynaptic organization of axo-axonic synapses in layer 2/3 of the somatosensory cortex of mice. We found that the gephyrin sub-synaptic domains of GABAergic axo-axonic synapses formed onto the AIS showed a distinct arrangement to those formed onto dendrites, suggesting specificity at the subcellular level. Increasing network activity in the somatosensory cortex using chemogenetics resulted in a decrease in sub-synaptic cluster volume in axo-axonic synapses onto the AIS, but not in axo-dendritic synapses. The reduction in gephyrin cluster volume reflected functionally weaker axo-axonic postsynaptic currents measured by electrophysiology in mouse acute brain slices. We propose that network activity in vivo can modulate the nanoscale arrangement of synapses and serves to fine-tune the strength of connections in the brain.