Multiple brain regions are involved in integrating reward to drive action selection, with rich representations of reward history in the striatum, retrosplenial cortex, and frontal areas. A major challenge in dissecting the circuits that govern reward-guided behavior is the existence of multiple strategies for reward maximization. For instance, in dynamic environments, mice can engage in both model-free behavior, where they update action values from trial to trial, and inference-based learning, where they use an internal model to infer the current world state. These two modes are challenging to distinguish, and can even intermix within training sessions, complicating studies of neural mechanisms that rely on session-averaged activity. Here, we tackled these problems by developing a computational approach to characterize dynamic shifts in behavioral strategies. We first simulated the choice sequences of model-free and inference-based agents, and built decoders of their underlying strategy using features of the choice transition around the block switches. We built on this analysis with a new state-space method, block Hidden Markov Model, which infers the hidden state that governs the behavior in each block of trials. Our analysis revealed a diverse mixture of both model-free and inference-based strategies even in expert animals, with an increased reliance on inference-based behavior with training. We used 1-photon widefield imaging to investigate how mesoscopic cortical activity varies with the inferred hidden state. We found that reward encoding is strongly state-dependent: reward is weakly encoded in the disengaged state, transiently encoded in the model-free state, and persistently encoded in inference-based learning. Activity in diverse cortical regions, including the somatosensory, motor, frontal and visual areas, showed different patterns of correlation with reward in each mode. Our results suggest distinct neural mechanisms that underlie different modes of dynamic foraging, and highlight the importance of hidden states in the dissection of reward circuits.