ePoster, COSYNE 2023
Neural representation and predictive processing of dynamic visual signals
Pierre-Étienne Fiquet, Pierre-etienne Fiquet and Eero Simoncelli
New York University / Flatiron Institute; Center for Neural Science; New York University / Flatiron Institute; Neural Science
- Montreal, Canada
- Computational Neuroscience, Computer Vision and
Poster
Abstract
All organisms make temporal predictions, and their evolutionary fitness level generally scales with the accuracy of these predictions. In visual perception, observer motion and continuous deformations of objects and textures imbue our visual input with distinct temporal structures, enabling partial prediction of future inputs from past ones. Inspired by recent hypotheses that primate visual representations support prediction by ``straightening'' the temporal trajectories of naturally-occurring input (Henaff et. al., 2019), we formulate an architecture for image representation that facilitates prediction by linearizing the temporal trajectories of frames of natural video. To facilitate this goal, we note that many deformations can be described as linear advances in the phase of a complex-valued representation. The most well-known example is that of the Fourier transform, whose complex coefficients have constant amplitude and shifting phase under translational motion, but the concept generalizes to all compact commutative Lie groups. We train a network to optimize next-frame predictions over large natural video datasets and show that it achieves performance on par with (or better than) that of traditional motion compensation and conventional deep networks while remaining interpretable and fast. The learned filters come in conjugate phase-shifted pairs and are selective for orientation and scale, which is reminiscent of the models used to describe selectivity of primary visual cortex neurons. Our results demonstrate the potential of a principled video processing framework for modeling visual processing and eventually linking behavioral performance with neural computations.
Topics
- temporal predictions, search World Wide
- visual perception, search World Wide
- natural video, search World Wide
- temporal trajectories, search World Wide
- image representation, search World Wide
- Fourier transform, search World Wide
- complex-valued representation, search World Wide
- Lie groups, search World Wide
- motion compensation, search World Wide
- primary visual cortex, search World Wide
Details
- Session
- Poster Session I