Perceptual decision-making involves interpreting sensory information to guide actions. Incoming sensory information must often be assessed in the context of other relevant information held in working memory. Despite this interdependence, the neural basis of working memory and perceptual decision-making have largely been studied separately. Thus, how the brain flexibly combines sensory information with information in memory to form decisions remains unknown. To study this interaction, we developed a novel behavioral task in which monkeys hold a set of color images in working memory as context, and then use that context to make decisions about new color images. By parametrically varying the similarity between context images and decision images on a trial by-trial basis, we examined how decisions result from the flexible integration of sensory and mnemonic information. Behavioral analysis revealed that the monkey’s choices and reaction times were influenced by the interaction between the difficulty of both the memory and the perceptual components, indicating they jointly affect decision-making processes. To investigate the neural basis of this interaction, we recorded neural activity from dorsolateral prefrontal cortex (dlPFC), a region with established neural correlates of working memory and decision making. We found that working memory and decision-related information were encoded by largely non-overlapping neural populations. When monkeys held context images in memory, the neural population represented not only the identity of individual images, but also the discriminability between them. When the decision images were presented, dlPFC neural dynamics reflected the influence of both mnemonic information and sensory input on evolving decisions. Our study provides new insights into how the prefrontal cortex combines remembered and perceived information to support flexible behavior.