The brain forms cognitive maps of relational knowledge, an organizing principle thought to underlie our ability to generalize and make inferences. Such map-like representations may facilitate goal-directed behavior by enabling generalization of information across related states. However, it is not fully understood what neural and computational mechanisms give rise to map-based generalization, nor how a relevant map may be selected when a stimulus is embedded in multiple relational structures. Here, we find that both spatial and temporal cognitive maps influence reward generalization in a choice task, where spatial location determines reward magnitude. Employing Gaussian Process regression to model reward generalization, we find corresponding hippocampal map representations of not only spatial relationships, but of temporal relationships as well, providing novel neural evidence for hippocampal map-based reward generalization. As the task progresses, we observe a strengthening of the spatial map and a weakening of the temporal map in the hippocampus, which is reflected behaviorally in participants’ choices becoming more influenced by the spatial map over time. We find that this change in both behavior and neural representations are driven by a signal in orbitofrontal cortex (OFC) which represents the degree to which an observed reward is consistent with the spatial rather than the temporal map, and which updates hippocampal representations accordingly. In the end, the participants whose decisions become more influenced by the spatial map as the task progresses display the largest increase in the strength of the spatial map representation in the hippocampus from before to after the choice task, an effect which is mediated by the signal in OFC. Overall, our work demonstrates computationally and neurally how hippocampal cognitive maps, in conjunction with a signal from OFC, are used and updated flexibly for inference and reward generalization.