Observation of neural firing patterns can constrain theories for the types of activity patterns that the brain uses to guide behavior. However, directly perturbing neural activity, ideally with great specificity, is required to causally test any particular theory. We combined two-photon (2p) imaging and cellular resolution optogenetic photostimulation1 to causally test how neural activity in the mouse primary visual cortex (V1) is read out to detect visual stimuli. Both an unweighted mean readout or a weighted read out of visually sensitive neurons could support detection, but the weighted read out is more optimal. However, contrary to expectations, targeted activation of highly visually sensitive neural ensembles did not preferentially modify behavior compared to random ensembles, contradicting a longstanding hypothesis for how neural activity drives stimulus detection. Instead, the main predictor of a targeted neural ensemble’s impact on perception was its effect on network activity. This argues that downstream regions summate V1 activity without preferentially weighting more informative neurons to make sensory detection decisions. Comparing mouse behavioral performance to decoding models of neural activity confirmed that mice employ this simple, albeit suboptimal strategy to solve the task. This strategy might promote generalization and be more ethologically advantageous. This work challenges conventional notions for how sensory representations mediate perception and demonstrates that neural perturbations are key for informing and constraining models of how neural activity drives behavior.