The power of deep neural networks for pattern recognition tasks lies in their universal approximation abilities, and their theoretical expressivity is well-studied. Practical expressivity, however, is known to lag behind theoretical expressivity. Network performance achieved in practice is determined by a set of inductive biases such as architectural constraints (e.g. residual connections). In this work, we introduce weakly coupled residual recurrent networks (WCRNNs) in which the residual connections result in well-defined Lyapunov exponents. This allows for studying properties of their dynamics and fading memory. We investigate how the residual connections of WCRNNs influence their performance, network dynamics, and memory properties on a set of benchmark tasks (adding problem and sequential MNIST).We show that several distinct forms of residual connections yield effective inductive biases that result in increased network expressivity (see figure). In particular, those are residual connections that (i) result in network dynamics at the proximity of the edge of chaos, (ii) allow networks to capitalize on characteristic spectral properties of the data, and (iii) result in heterogeneous memory properties.Taken together, our analyses not only show the influence of RNN dynamics on the practical expressivity, but also allow for giving functional interpretations of some hallmark properties of cortical dynamics such as criticality, and oscillatory dynamics.