Understanding how sensory neurons extract relevant information from natural scenes is a major challenge in neuroscience. Probing sensory systems with simple stimuli give limited insights on natural scene processing, and probing selectivity during natural scene stimulation is difficult. Even in the retina, is not clear how Retinal ganglion cells (RGC) extract specific features from natural scenes and send this information to the brain. Many studies using simple, artificial stimuli have shown responses to local light increase (ON-responses), and/or decrease (OFF), but it is unclear if this selectivity is maintained when processing natural stimuli. Other works tried learning non-linear models to predict RGC responses to natural stimuli, but are hard to interpret. Here we address these issues using a novel perturbative approach that takes the best of these two strategies by probing selectivity with perturbations added to natural scenes. We stimulated mouse and axolotl RGCs with natural images, adding small checkerboard-like perturbations on top. We found that single RGC can be selective to light increments for a perturbed natural image, and to light decrements when the same perturbations are added to another. RGCs can thus switch selectivity from ON to OFF depending on the natural context. We designed a convolutional neural network model to explain these changes, and mapped it to specific retinal circuits. Pharmacological experiments and modeling showed that ON/OFF selectivity changes were due to the non-linear combination of different retinal pathways. Finally, using dimensionality reduction and gradient field representations, we demonstrated that this strong context dependence is compatible with a robust computation of a more abstract feature: contrast. Our perturbative approach thus uncovers neuronal selectivity to more complex features than initially thought during natural scene stimulation and could be applied in other sensory systems to refine models or test hypotheses about what features are extracted from the sensory input.