The ability to encode visual motion information is an essential prerequisite for organisms to navigate their surroundings. For example, animals need to detect whole-field motion in order to stabilize themselves within their environment. While it is known that some features of naturalistic motion such as its directionality are already extracted in the retina, it is still unclear whether all the necessary cues are already available at these early stages. Here we take advantage of the optomotor response (OMR) behavior in the larval zebrafish to study how these animals respond to whole-field motion moving at different velocities. Through a series of behavioral and functional imaging experiments, we show that the zebrafish OMR is tuned to the temporal frequency of the stimulus, providing support for some of the fundamental properties of classical models for motion detection. Moreover, we also observe that certain bout features modulated by temporal frequency are set at the moment of triggering the behavioral output rather than controlled online during the behavioral response, suggesting that the action selection mechanism is more complex than previously thought. Through whole-brain light sheet experiments we identified neural populations in the optic tectum and the pretectum especially suited to encode stimulus motion velocity. These findings provide a deeper understanding of the encoding of visual motion representations in the zebrafish brain and the sensorimotor transformations required to convert the visual input into action.