Normative neural network models provide a valuable tool for answering principled ‘why’ questions about neural structure, function, and organisation from an optimization perspective. However, typical approaches often compare models without controlling for differences in architecture, training dataset or model objective. As a result, it is often difficult to make strong conclusions relating variation in model performance to a given characteristic. Here we ask what factors tend to make models predict neural responses most effectively?To investigate this, we explored how hyperparameter choice impacts the capacity of the models to predict the single-unit responses of neurons in ferret primary auditory cortex (A1) to natural sounds. A large number of models were trained while varying the dataset, model architecture and model objective. The activity of each model in response to natural sounds was then regressed to predict the firing rate of A1 neurons. Finally, models were analysed to characterise how different model properties relate to neural prediction performance.While many different models performed above the linear-nonlinear baseline, model architecture, training dataset and objective all had distinct and marked impacts on the capacity to predict A1 responses. As expected, the similarity of model receptive fields to A1 neurons was strongly correlated with neural prediction performance. Most notably, we found that models which were better able to generalize to novel downstream tasks (i.e. showed better transfer learning) were also better at predicting neural responses. Thus, hyperparameter choices which encouraged the models to learn more general representations tended to better predict A1 responses.