Neurons in primary Visual Cortex (V1) have a receptive field (RF) or "center", the region in visual space in which appropriate stimuli drive the neuron to fire. Stimuli surrounding the RF modulate these responses, a phenomenon known as contextual modulation. Here we consider 3 types of contextual modulation: i) classical surround suppression: the response of cells in V1 decreases when the stimulus size is increased beyond the RF size; ii) inverse surround suppression: neurons in layer L2/3 –but not layer L4– of mouse V1 respond to a lack of contrast in their RF when the surround is filled with a drifting grating (‘hole’ stimulus), and this response shows tuning for the size of the hole similar to i) and iii) surround facilitation: the response of cells in V1 increases when a grating stimulus in the RF is presented together with an orthogonal surround. We devise a large-scale stabilized supralinear model of rate units representing L2/3 cells, receiving recurrent, bottom-up (L4) and top-down (higher visual areas HVA) inputs. The model is endowed with anatomically realistic length scales and physiologically constrained input patterns and is able to account for all three contextual modulation phenomena. Given the recorded inputs to L2/3 in [Keller-Nature2020], we develop a conceptual model to analytically compute the firing rate profile across layer L2/3. We shed light on the mechanisms and different factors that contribute to classical and inverse surround suppression. We predict the specific role of Somatostatin interneurons in inverse response and propose a link between inverse response and surround facilitation. We then present a ‘full’ model with three interneuron subclasses showing the robustness of the identified mechanisms. The full model is built to match recordings during the classical stimulus condition and generalizes to reproduce inverse size-tuning curves.