Our intuition suggests that when a movie is played in reverse, our perception of motion at each location in the reversed movie will be perfectly inverted compared to the original. This intuition is also reflected in classical theoretical and practical models of motion estimation, in which velocity flow fields invert when inputs are reversed in time. However, this symmetry of motion perception upon time reversal is broken in real visual systems. We designed a set of visual stimuli to investigate time reversal symmetry (TRS) breaking in the fruit fly Drosophila’s optomotor rotation behavior. We discovered a suite of new stimuli with a wide variety of properties that can uncover broken TRS in fly behavioral responses. We then trained neural network models to predict the velocity of scenes with natural and artificial contrast distributions. Training with naturalistic contrast distributions yielded models that broke TRS, even when the training data itself was time reversal symmetric. We show analytically and numerically that the breaking of TRS in the model responses can arise from contrast asymmetry in the training data and other features of the contrast distribution. Furthermore, shallower neural network models can exhibit stronger symmetry breaking than deeper ones, suggesting that less flexible neural networks may be more prone to TRS breaking. Overall, this research identifies conditions for breaking TRS in animal motion percepts and computational models for motion estimation. The results show how this surprising feature of biological motion detectors could arise from constrained optimization in natural environments.