Spontaneous activity in young ferrets (p22) is organised into modular, low-dimensional patterns across multiple cortical areas [1]. In the following two weeks (p22-43), these activity patterns undergo a transition similar in all areas, and show a gradual increase in dimensionality in most areas, suggesting an expansion of representational capacity [2]. However, it is unclear what the functional role of having low-dimensional activity patterns at the initial stage of development is, since one would expect the limited dimensionality to restrict representational learning. In this work, we investigated the functional role of an initial low dimensionality of sensory representations early development and hypothesised that it facilitates the formation of well-structured receptive fields. In order to understand the implications of this low dimensionality on representational learning, we focused on the well-studied visual system, and studied self-supervised representation learning in Autoencoder (AE) models. We proposed the developing autoencoder (DAE): a novel training method that allows a gradual increase of bottleneck size throughout training. In this way, these models initially learn restricted representations, which can then become higher-dimensional after certain training epochs. By comparing DAE models with standard AE models, we investigated the implications on reconstruction performance and receptive field formation. Firstly, we found that pre-training at even smaller bottleneck sizes improves subsequent learning speed, suggesting features learned during early stages are helpful for later learning stages. In addition, by comparing inputs that maximise the activation of bottleneck units, we found that a standard AE learns complex visual features with high spatial frequencies, while the DAE learns a combination of simple and complex features with high and low spatial frequencies, more in line with biologically observed receptive fields in V1. Our results suggest that the observed delayed dimensionality expansion facilitates the formation of more diverse receptive fields.