A venerable tradition in neuroscience seeks to understand sensory processing, and in particular vision, through unsupervised learning of natural statistics. Generative models have provided invaluable insights into the way neural response statistics of low level vision (including nonlinearities in response means, variability, oscillations) is shaped by probabilistic inference. However, progress has been hampered by the limited capabilities of generative models to learn both nonlinear and hierarchical representations of natural images. Here we harness the inspirations coming from neuroscience to develop a novel flavor of Variational Autoencoders, a class of models that is capable of performing learning and inference, of/in nonlinear generative models. Key to the proposed hierarchical generative model, TD-VAE (Top-Down Variational Autoencoder), is a formulation which builds on the top-down feedback connections between cortical processing stages. We show how inductive biases contribute to shaping the representations emerging in V1 and V2 of the visual cortex, including Gabor-like filters in V1 and texture selectivity in V2 when trained on natural image patches. The model reproduces a number of earlier experimental observations about the interdependence of the activities in V1 and V2. These include progressive compression of images along the hierarchy of the ventral stream, differential sensitivity of V1 and V2 mean responses to the manipulations of high-level statistics. Further, we use TD-VAE to demonstrate that effects that were implicated in top-down influences in V1, such as stimulus-statistics dependent noise correlations and illusory contours, are natural consequences of hierarchical probabilistic inference in TD-VAE. Our study provides a tool that can be used as a starting point for the explorations of the visual cortex through a nonlinear hierarchical generative model for natural images.