Model metamers complement existing benchmarks of biological and artificial neural network alignment

The proliferation of deep artificial neural network models has given rise to widespread interest in comparing such models to biological sensory systems. Model metamers – stimuli that produce the same responses in a model as a natural stimulus – have been proposed as a powerful test of the extent to which a model replicates the invariances of a biological sensory system. However, it has remained unclear whether discrepancies evident from metamers are the same as those identified by other model comparison benchmarks, and whether modifications that have been introduced to address other well-known model discrepancies also serve to mitigate metamer-related discrepancies. We considered three recently introduced benchmarks for model comparison and asked whether performance on these benchmarks predicts the extent to which a model’s metamers are recognizable to humans. First, we examined performance on two sets of image distortions that show benefits of training models on larger and more diverse image sets. Second, we assessed whether a model’s over-reliance on texture cues (a commonly cited difference between humans and neural networks) relates to model metamer recognition. Third, we evaluated model predictions of neural responses in V1, V2, V4, and IT. In each case, performance on the benchmarks was at best weakly related to the human-recognizability of a model’s metamers. The results show that metamers are complementary to other popular model comparison benchmarks, and identify distinct failure modes of current neural network models. More generally, our work highlights the uses of model-driven stimulus synthesis to better understand and refine models of the brain.