Biological neuronal networks exhibit hallmark features such as oscillatory dynamics, heterogeneity, modularity, and conduction delays. However, it has remained unclear to what extent these serve computational purposes. Inspired by physiological findings, we simulate recurrent networks of damped harmonic oscillators (DHO) in which one network node represents the aggregate activity of an underlying E-I circuit such as a cortical column (HORNs, see figure), at the same time reducing single-node oscillatory dynamics to its most basic form. We successively endow the networks with additional biological features such as heterogeneous node frequencies, scattered conduction delays, and multilayer architectures. Despite their conceptual simplicity, the networks were found to be able to reproduce a surprising number of fundamental neurophysiological findings while at the same time outperforming non-oscillating RNN architectures in learning speed, task performance, parameter efficiency, and noise tolerance in standard pattern recognition benchmarks (MNIST hand written digit recognition, spoken word recognition). Our analyses uncover the powerful computational principles realized in such networks (such as feature detection and coding through resonance and stimulus coding by means of waves) and allow us to give plausible a posteriori functional interpretations for many fundamental anatomical and physiological features of cortical networks such as the dynamics of synchronisation and desynchronisation, the heterogeneity of preferred oscillation frequencies and conduction delays, the context dependence of receptive fields, and multilayer hierarchies. Lastly, we show how our model enables biologically plausible unsupervised Hebbian learning and discuss how this architecture could inspire novel analog computing devices.