The vestibulo-ocular reflex (VOR) stabilizes gaze by rotating the eyes into the opposite direction of the head. Optimal VOR function requires continuous calibration of the vestibular-to-motor transformation using retinal image slip as the sensory error; an operation that largely depends on the cerebellar flocculus. Here, we investigate how compensatory eye movements develop when retinal slip signals are largely absent, using mouse models with a defect in retinal motion detection. Congenital defects in the retinal photoreceptor-to-bipolar cell synapse eliminate the retinal ON-pathway and impair transmission of retinal slip to floccular climbing fibers. We show that this condition not only leads to an OKR defect, but also to a highly nonlinear disruption of the VOR, resulting in anti-compensatory responses at low stimulus velocities. Still, the VOR responses are closely replicated by a relatively simple computational model that includes an indirect VOR pathway with a saturating static nonlinearity and band-pass filter. We demonstrate with lesion experiments that the nonlinear pathway resides in the cerebellar flocculus and confirm our model predictions using extracellular recordings of Purkinje cell activity during vestibular stimulation. During anti-compensatory VOR responses, simple spike activity is inversely modulated compared to wildtype mice. While complex spike modulation is virtually absent during low-velocity OKR, nonvisual modulation during VOR in darkness is still intact, showing band-pass filter characteristics predicted by the computational model. We conclude that defects in visual motion detection lead to a highly nonlinear VOR as a consequence of life-long cerebellar maladaptation due to persisting extraretinal signals on floccular climbing fibers.