Topic: regularised gate modulation

The success of neural networks derives from the fact that they can learn useful representations of observed inputs. However, the dynamics of when and how fast a network will discover good representations are not well understood, even when network training is governed by gradient descent. The potential complexity of learning dynamics is particularly apparent in so called insight moments, when useful task representations are discovered suddenly following a delay during which no learning is noticeable. Such insights are commonly observed in animals and humans, where they are often taken to refl