In recent years electrophysiological studies have profited from the use of multivariate pattern analysis methods. However, these often rely on outdated classification algorithms. Deep neural networks provide high performance at the cost of interpretability of learned features because of black box approaches. In this project, we evaluated how the interpretability of networks for electroencephalography (EEG) data is affected by the network architecture and the method for feature extraction and visualisation. To this aim, we used two convolutional neural network architectures: (1)ResNet, a residual network, and (2)EEGNet which leverages spatiotemporal properties of EEG signals. We trained these networks to discriminate single-trial EEG responses to three different visual stimuli (visual dataset) and two sounds (auditory dataset). We then extracted and visualised learned features with two gradient-based techniques: Saliency and gradient-weighted activation maps(GradCam). Results show that the two networks performed at a similar level (Fig.1A). Classification accuracy of EEGNet and ResNet in an independent test dataset was 0.52±0.02 and 0.53±0.02 respectively on the visual dataset and was 0.63±0.02 and 0.58±0.01 respectively on the auditory dataset. The spatial distribution of learned features by EEGNet was closer to EEG topographic responses. However, the timing and distribution of electrodes with high importance differed between the two visualisation techniques (Fig.1B). Our results suggest that the two neural networks provide similar classification performance but they learn different aspects of the data. Our results call for careful consideration of network architecture and feature visualization techniques to improve interpretability and for deep learning in EEG research.