Deep learning based methods are considered a viable alternative for a quick and reliable whole-brain MRI segmentation. This work aims at a quantitative evaluation of the performance of DeepSCAN and FastSurferCNN, two different deep-learning based segmentation approaches, in comparison to the well studied Freesurfer reconstruction. Going beyond assessment based on Dice indices, the purpose is to test the reliability and robustness of the resulting cortical surface reconstruction. It was used 3D T1-weighted MR images from four open-access databases, each one selected to test different aspects of the segmentation: (i) The “Phantom of Bern”, a dataset of same-session rescans obtained from two healthy subjects using different imaging contrast; (ii) A synthetic dataset with a ground truth constructed by systematically reducing the cortical thickness of a brain to a known value; (iii) The AOMIC-ID100 dataset which has three equivalent runs; (iv) Subjects from OASIS3 that originally failed in Freesurfer reconstruction. The surface reconstruction was done using Freesurfer pipeline but changing the initial segmentation accordingly. The results shows different contrast dependency between the segmentation methods. Regarding the synthetic dataset, DeepSCAN shows better performance matching the ground truth expectation and presenting no failed surfaces. The data also suggests that the synthetic dataset failed to capture the natural variability of the human brain. Both deep-learning based segmentation presents a performance compatible with Freesurfer regarding the repeatability test using the AOMIC-ID1000 dataset and were able to successefuly reconstruct brains failed by Freesurfer’s pipeline.