Many animals use visual motion cues to track and pursue small, fast-moving prey or to track conspecifics. The pursuit of small visual targets is a hard problem and understanding the underlying neural circuits will inform similar studies in other animals and open avenues for bio-inspired robotics. In dragonflies and hoverflies, the target-sensitive neurons found in the optic lobes are known as small target motion detectors (STMDs). STMDs are believed to be presynaptic to the target selective descending neurons (TSDNs) that are in turn presynaptic to motor neurons. In TSDNs, the response to targets is modulated by background motion. Depending on whether the background motion is syn- or contra-directional to the target motion, the TSDNs either suppress or amplify the STMD response. However, the neural circuitry behind how the TSDN operates and how the TSDN output is involved in target pursuit is not clearly understood. To explore the neural circuits underlying target pursuit, we first performed extracellular recordings from TSDNs in hoverflies experiencing controlled visual stimulation. Then, we fitted different circuit models consisting of a model of the STMD circuit, known optic flow-sensitive neurons (LPTCs) and the TSDN itself, to the electrophysiology data we obtained. We present a model of the TSDN circuit from a photoreceptor to premotor level. The model explains the mechanism of how background motion leads to amplification and suppression of responses described above. The model does not involve learning and hence suggests that the TSDN circuit is implementing an innate optomotor reflex. As the model is quite parsimonious and only depends on commonly observed simple filters it might be conserved across other species.