Retinal scene statistics for freely moving mice

Mice sample the visual scene by moving their head and eyes. However, little is known about how eye and head position are integrated into visual processing during natural movement, because studies of visual processing are generally performed during head fixation. To address this, we analyzed the retinal input encountered during free exploration by measuring the mouse’s visual scene (“worldcam”) and eye/head position using head-mounted cameras and an inertial measurement unit (IMU). Based on these measures, we trained a neural network to estimate the retinal input by gaze-correcting the worldcam video with an affine transformation. We then extracted several spatiotemporal features (e.g., luminance, contrast, optic flow, and Difference of Gaussian (DoG) entropy) of the visual input during free exploration and analyzed how their distributions were affected by gaze-shifting and compensatory eye/head movements. We found that compensatory eye movements decreased optic flow (as expected), but increased overall luminance as well as DoG entropy in the lower half of the visual field. To discover which visual features drove gaze-shifting eye movements, we used VGG-16 to extract convolutional features from video frames immediately preceding a gaze shift and predicted the horizontal angle of the saccadic endpoint. Our model could account for 47% of the variance in saccade targeting. A saliency map analysis of the deep net revealed that the pixels that were most predictive of saccadic endpoints were often localized to nearby objects on the ground consistent with navigation and obstacle avoidance, or to the upper visual field consistent with scanning for predators. These results may provide insight into how mice use their eyes to sample the visual scene, with implications for visual processing beyond head-fixed preparations.