Topic: network expressivity

The power of deep neural networks for pattern recognition tasks lies in their universal approximation abilities, and their theoretical expressivity is well-studied. Practical expressivity, however, is known to lag behind theoretical expressivity. Network performance achieved in practice is determined by a set of inductive biases such as architectural constraints (e.g. residual connections). In this work, we introduce weakly coupled residual recurrent networks (WCRNNs) in which the residual connections result in well-defined Lyapunov exponents. This allows for studying properties of their dynam