In natural speech, the harmonic pattern of a speaker’s fundamental frequency (f0) and its higher harmonics elicits a neural response at the fundamental frequency which can be captured by electroencephalography (EEG) or magnetoencephalography (MEG), termed speech-FFR. While EEG measurements have found a subcortically evoked early neural activity to this speech-FFR, MEG studies have recently emphasized involvement of cortical sources as well. Here, we focused on disentangling the sources of the neural response at the fundamental frequency of speech using EEG-based source localization. We therefore analysed EEG recordings of subjects who listened to audiobooks read by a single speaker. We extracted two features from the speech signal, 1) the carrier signal around f0, and 2) the high gamma envelope modulation at higher harmonics. For the second stimulus, we used three different processing strategies to compute the high-frequency spectra, here named gammatone, bandpass and inner-ear model. We subsequently trained linear regression models to obtain Temporal Response Functions (TRF) for both features and applied inverse source analysis on the resulting coefficients. For all three models, we found major midbrain activity with weaker brainstem contributions. Interestingly, for the bandpass and inner-ear model, minor cortical activity could be identified at different latencies elicited by the fundamental waveform and the envelope modulations. Our results show that both subcortical and cortical activity can be captured by EEG-based source localization of the speech-FFR at the fundamental frequency of speech while still subcortical sources largely dominate the EEG-captured speech-FFR.