Unlike conventional deep neural networks, the human brain has a myriad of direct connections from subcortical nuclei to all cortical areas [1,2]. These enable fast information transfer and facilitate hierarchical compositionality but have not yet been explored in artificial systems [3]. In this work, we present the Shallow Hierarchical Artificial Neural Network (SHANN), a novel brain-inspired architecture with shallow connections from the input to all the hierarchical processing layers. Like conventional deep networks, SHANNs can achieve high levels of abstraction in more hierarchical layers. However, the additional direct connections from the input to all hidden layers can facilitate faster training and transfer of information. We show that SHANNs outperform both shallow and deep networks in reconstruction and classification tasks. Evaluations of the trained models reveal that SHANNs perform hierarchical compositionality to combine information from different levels of abstraction. We believe the shallow hierarchical architecture can result in valuable improvements to artificial neural networks and help bridge that gap between neuroscience and machine learning.