Visual cortical neurons are richly inter-connected by within-area, and feedforward and feedback across-area circuits. How these different circuits contribute to visual function remains poorly understood. We performed simultaneous recordings from hundreds of neurons in V1 and V2 in anesthetized macaque monkeys. Using a latent variable model, we identified the feedforward, feedback, and within-area signals that contributed to the measured responses. The identified feedforward and feedback signals contributed equally to explaining activity in the two areas, consistent with their equal anatomical prominence. Additionally, across-area signals were stronger between groups of neurons with retinotopically aligned receptive fields compared to those that were misaligned. We next studied how the contribution of these signals changed with visual input. First, we found that areas decouple during spontaneous activity, as evident by minimal across-area signals; only with stimulus drive were feedforward and feedback signaling evident. Second, the proportion of within- to across-area signaling in V1 was higher for drifting gratings compared to natural movies, whereas in V2 there was a slight trend in the opposite direction, indicating that the spatial features of visual input may affect the strength of recurrent signaling. Finally, we tested how spatial and temporal context of visual stimulation affected signaling. Using center-surround grating stimuli, we found that the strength of feedforward signaling was reduced when the surround stimulus was informative about the center. Temporally, a mismatch between the current and preceding stimulus enhanced feedforward signaling. Together, our results show that the functional prominence of across- and within-area signals are consistent with anatomical constraints, but these signals are flexible and depend on the spatiotemporal features and context of visual input.