The best current models of visual cortical neural responses rely on internal representations of large deep neural networks (DNNs) trained for object recognition. However, the inner workings of these DNN models are mostly uninterpretable due to their massive size—tens of millions of parameters. Do we need such large models in the first place? In this work, we focused on modeling visual cortical neurons in macaque mid-level visual area V4. We used ensemble learning and closed-loop experiments with active learning to train a DNN model to accurately predict V4 responses. However, our model had 90 million parameters—too many to interpret. We then leveraged machine learning techniques of distillation and pruning to identify compact models that were 1,000x smaller than the large model but still as predictive. We causally tested our compact models by using them (1) to adaptively synthesize images to maximize V4 responses and (2) to slightly perturb images to yield large changes in V4 responses (i.e., adversarial images). We found that the compact models preferred a large variety of oriented edges, curves, textures, and colors. One prominent preference was for small dots; we ran additional experiments to probe the properties of these “dot detectors” in V4. We then analyzed the compact “dot detector” model to uncover the necessary and sufficient computations needed to build such a detector. Overall, we propose a general approach to identify highly-predictive, interpretable models; we used this approach to find compact models of V4 neurons whose size is substantially smaller than previously thought.