Critical Learning Periods for Multisensory Integration in Deep Networks

We show that the ability of a neural network to integrate information from diverse sources hinges critically on being exposed to properly correlated signals during the early stages of learning. Interfering with the learning process during this initial stage can permanently impair the development of a skill, both in artificial (Achille et al., 2019) and biological systems where the phenomenon is known as a critical learning period (Wiesel, 1982). Critical periods have since been described in many different species and sensory organs: for example, barn owls originally exposed to misaligned auditory and visual information cannot properly localize prey (Knudsen & Knudsen, 1990). We show that critical periods arise from the complex and unstable early transient dynamics, which are decisive of final performance of the trained system and their learned representations. In fact, using a simple linear model, we show that deep linear networks exhibit critical learning periods for multi-source integration, while shallow networks do not. We also find the existence of critical learning periods for deep multi-source networks on image classification tasks, in the presence of a temporary weak or uncorrelated source. To better understand how the internal representations change according to disturbances or sensory deficits on realistic tasks, we introduce a measure of source sensitivity, which allows us to track the inhibition and integration of sources during training. Our analysis of inhibition suggests cross-source reconstruction as a natural auxiliary training objective, and indeed we show that architectures trained with cross-sensor reconstruction objectives are remarkably more resilient to critical periods. Our findings shed light on how critical learning periods may arise from competition between sources, which begins during the early stages of development in deep artificial and biological networks.