Biological circuits adapt their behaviour through experience and ongoing learning in a non-trivial manner. Experimental evidence shows differing neuronal population responses to familiar and novel stimuli [1, 2], indicating that neural circuits operate in distinct processing modes depending on stimulus familiarity. In layer 2/3 of mouse primary visual cortex (V1) specifically, populations of pyramidal and vasoactive-intestinal-peptide-expressing (VIP) cells show pronounced responses to novel visual stimuli (novelty responses) and weaker responses to familiar ones. Although experimental and theoretical work has suggested different origins for these differing responses [3-6], the specific plasticity mechanisms underlying the adaptation to familiar stimuli have not yet been fully characterised. Here, we set up a data-constrained recurrent neural network model with three interneuron sub-types [7, 8] , which is adaptively shaped through long-term plasticity to a set of stimuli. Using linear response theory [9, 10], we identify necessary conditions for circuit design to replicate experimentally observed cell-type-specific response profiles to both contextual and absolute stimulus novelty, as well as to temporal expectation violations during stimulus omissions. Our work provides a mechanistic explanation of how cortical circuits adjust their responses to stimulus expectations and offers testable predictions for previously unobserved parvalbumin-expressing (PV) interneuron responses in this context.