What information do cortical neurons along the visual hierarchy encode for? The classic view that neural populations in the visual cortex preferentially encode responses to visual stimuli has been strongly challenged by recent experimental studies. For example, a large fraction of variance in V1 responses in rodents can be attributed to behavioral state1, trial-history2, etc. Here, we present a comprehensive experimental and theoretical study demonstrating that the cortical visual hierarchy differentially encodes the temporal context and expectation of visual stimuli. We measured layer-specific neural responses to expected and unexpected sequences of natural scenes across three visual areas using in vivo 2p imaging in mice: the primary visual cortex (VISp), the posterior medial higher order visual area (VISpm), and retrosplenial cortex (RSP). We found that all three areas predominantly encode for the temporal context in which the images were presented. Information about image identity was not present in neural populations when images were presented in randomized order. Only when presenting images within long repeated sequences, VISp and VISpm, but not RSP, encoded image identity as well. We found that the conjunctive encoding of temporal context and image identity was modulated by the emergence of expectation about sequential events. We compared responses to expected vs unexpected image sequences, the latter one comprising an occasional oddball image replacing the last image in a familiar sequence. We found enhanced responses to oddball images when they disrupted the expected sequence order. The oddball response was strong in VISp and VISpm but weak in RSP. In the latter, we found evidence for predictive coding where the oddball response recapitulated the identity of the missing image. Our results establish temporal context and expectation as new encoding dimensions in the visual hierarchy and suggest that differential responses along the visual hierarchy instantiate a predictive coding mechanism.