The human visual cortex is modular, parcellated into a hierarchy of visual areas (V1, V2, V3, etc.) that abut each other in the brain. The areas are each retinotopically organized with several characteristing higher-level organizational features --- a notable example is that the polar angle of the retinotopic maps alternates (is mirrored) across area boundaries. Normative (task-trained) models of the dorsal visual stream capture important aspects of neural tuning, but typically only involve feedforward connections and already assume a hierarchical organization of a discrete set of areas. They also do not explain the spatial organization of these areas in the brain. On the other hand, recent work with self-organizing maps has shown that it is possible to explain some of the spatial aspects of the organization of visual areas through multi-scale spatial relationships alone with requiring complex feature tuning relationships. Complementing this approach, here we take a developmental perspective in which the visual cortical connections are grown from simple bottom-up rules. These rules rely on an activity-dependent wiring process driven by spatial relationships implicit in the structure of retinal waves, and a synaptic pruning process dependent on wiring length, both grounded in biological data. Several features emerge as a result of the growth process: First, a discrete set of areas develops, with largely feedforward connections between them, defining a hierarchy. Second, each area exhibits retinotopy and characteristic mirror reversals in polar angle. The global eccentricity map is preserved. Third, connectivity is local: nearby neurons connect to nearby neurons in the preceding level of the hierarchy, together with a small fraction of recurrence. And lastly, receptive field sizes increase along the hierarchy. Altogether, this study demonstrates that many features of brain organization may arise not by direct optimization on particular tasks but as a consequence of low-level biophysical rules which unroll a cascade of developmental processes that determine structure and connectivity.