Repeat-rich sequence blocks are considered major determinants for 3D folding and structural genome organization in the cell nucleus in all higher eukaryotes. Here, we discuss how megabase-scale chromatin domain and chromosomal compartment organization in adult mouse cerebral cortex is linked, in highly cell type-specific fashion, to multiple retrotransposon superfamilies which comprise the vast majority of mobile DNA elements in the murine genome. We show that neuronal megadomain architectures include an evolutionarily adaptive heterochromatic organization which, upon perturbation, unleashes proviruses from the Long Terminal Repeat (LTR) Endogenous Retrovirus family that exhibit strong tropism in mature neurons. Furthermore, we mapped, in the human brain, cell type-specific genomic integration patterns of the human pathogen and exogenous retrovirus, HIV, together with changes in genome organization and function of the HIV infected brain. Our work highlights the critical importance of chromosomal conformations and the ‘spatial genome’ for neuron- and glia-specific regulatory mechanisms and defenses aimed at exogenous and endogenous retrotransposons in the brain