Zero-shot learning of decodable natural features from the retina using a U-net

Much of sensory neuroscience focuses on presenting stimuli that are chosen by the experimenter because they are parametric and easy to sample and are thought to be behaviorally relevant to the organism. However, it is not generally known what the relevant features are in complex, natural scenes. This work focuses on using the retinal encoding of natural movies to determine the presumably behaviorally-relevant features that the brain represents. It is prohibitive to parameterize a natural movie and its respective retinal encoding fully. We use time within a natural movie as a proxy for the whole suite of features evolving across the scene. We then use a task-agnostic deep architecture, to model the retinal encoding process and characterize its representation of ‘time in the natural scene’ in a compressed latent space. In our end-to-end training, an encoder learns a compressed latent representation from a large population of salamander retinal ganglion cells responding to natural movies, while a decoder samples from this compressed latent space to generate the appropriate future movie frame. We show that the retina is responding to texture and motion features that change over time. To our surprise, we find that the retina has a generalizable encoding for time in the natural scene: The latent space learned from one movie can be used to do zero-shot decoding (without retraining) of time in an unseen, novel movie. This finding suggests that the retina leverages feature representations that are common across natural movies.