Vision transformer models for predicting neural responses: insights and challenges

Arno Onken examines transformer models for predicting responses in the retina and visual cortex. These models learn complex neural-response patterns from large datasets and can transfer learned representations between settings, but fitting and interpreting them differs from linear–nonlinear and convolutional approaches. The talk discusses the opportunities and difficulties this creates for understanding sensory processing.

It includes ViV1T, a transformer trained on natural movies that revealed previously uncharacterised response properties in mouse primary visual cortex. The work illustrates how predictive models can generate hypotheses and guide animal experiments. Onken also considers the substantial data and computing requirements and the difficulty of interpreting what the models have learned.