Social behaviors result from processing social cues (sensory cues from a partner) and producing actions relative to both the other’s behavior and one’s own internal state. How these two (social sensory cues and internal states) come together to drive behavior on a moment-to-moment basis is not yet clear. Here we fill this gap focusing on social interactions in Drosophila melanogaster. During courtship, males chase females and sing; female responses to male cues serve as critical feedback signals that shape the male’s ongoing behavior (Coen et al. 2014). In addition, we know that internal states shape these interactions (Calhoun et al. 2019 and Deutsch et al. 2020). Although courtship in Drosophila has been studied for decades (Yamamoto and Koganezawa 2013) we still lack a comprehensive map of the elaborate closed-loop social interactions, in both males and females, that comprise the courtship repertoire. This has been an impediment in interpreting manipulations of neural activity. To address this issue, we built a new behavioral phenotyping system that captures fine-scale postural descriptors of freely-moving, interacting fruit flies, and we leverage a deep learning framework (SLEAP) for multi-animal pose estimation to produce a rich, complex behavioral dataset of >90 pairs of freely moving, courting fruit flies. Finally, we introduce a novel computational framework that integrates both social sensory cues and internal states on multiple timescales to determine how they shape social dynamics.