Many animals live in complex social groups. To survive, it is essential to know who to avoid and who to interact. Although naïve mice are naturally attracted to any adult conspecifics, a single defeat experience could elicit social avoidance towards the aggressor for days. The neural mechanisms underlying the behavior switch from social approach to social avoidance remains incompletely understood. Here, we identify oxytocin neurons in the retrochiasmatic supraoptic nucleus (SOROXT) and oxytocin receptor (OXTR) expressing cells in the anterior subdivision of ventromedial hypothalamus, ventrolateral part (aVMHvlOXTR) as a key circuit motif for defeat-induced social avoidance learning. After defeat, aVMHvlOXTR cells drastically increase their responses to aggressor cues. This response change is functionally important as optogenetic activation of aVMHvlOXTR cells elicits time-locked social avoidance towards a benign social target whereas inactivating the cells suppresses defeat-induced social avoidance. Furthermore, OXTR in the aVMHvl is itself essential for the behavior change. Knocking out OXTR in the aVMHvl or antagonizing the receptor during defeat, but not during post-defeat social interaction, impairs defeat-induced social avoidance. aVMHvlOXTR receives its private supply of oxytocin from SOROXT cells. SOROXT is highly activated by the noxious somatosensory inputs associated with defeat. Oxytocin released from SOROXT depolarizes aVMHvlOXTR cells and facilitates their synaptic potentiation, and hence, increases aVMHvlOXTR cell responses to aggressor cues. Ablating SOROXT cells impairs defeat-induced social avoidance learning whereas activating the cells promotes social avoidance after a subthreshold defeat experience. Altogether, our study reveals an essential role of SOROXT-aVMHvlOXTR circuit in defeat-induced social learning and highlights the importance of hypothalamic oxytocin system in social ranking and its plasticity.

The main pillars of social behaviors involve (1) mating, where males copulate with female partners to reproduce, and (2) aggression, where males fight conspecific male competitors in territory guarding. Decades of study have identified two key regions in the hypothalamus, the medial preoptic nucleus (MPN) and the ventrolateral part of ventromedial hypothalamus (VMHvl) , that are essential for male sexual and aggressive behaviors, respectively. However, it remains ambiguous what area directs excitatory control of the hypothalamic activity and generates the initiation signal for social behaviors. Through neural tracing, in vivo optical recording and functional manipulations, we identified the estrogen receptor alpha (Esr1)-expressing cells in the posterior amygdala (PA) as a main source of excitatory inputs to the MPN and VMHvl, and key hubs in mating and fighting circuits in males. Importantly, two spatially-distinct populations in the PA regulate male sexual and aggressive behaviors, respectively. Moreover, these two subpopulations in the PA display differential molecular phenotypes, projection patterns and in vivo neural responses. Our work also observed the parallels between these social behavior circuits and basal ganglia circuits to control motivated behaviors, which Larry Swanson (2000) originally proposed based on extensive developmental and anatomical evidence.

Social interactions such as mating and fighting are driven by internal emotional states. How can we study internal states of an animal when it cannot tell us its subjective feelings? Especially when the meaning of the animal’s behavior is not clear to us, can we understand the underlying internal states of the animal? In this talk, I will introduce our recent work in which we used male mounting behavior in mice as an example to understand the underlying internal state of the animals. In many animal species, males exhibit mounting behavior toward females as part of the mating behavior repertoire. Interestingly, males also frequently show mounting behavior toward other males of the same species. It is not clear what the underlying motivation is - whether it is reproductive in nature or something distinct. Through detailed analysis of video and audio recordings during social interactions, we found that while male-directed and female-directed mounting behaviors are motorically similar, they can be distinguished by both the presence of ultrasonic vocalization during female-directed mounting (reproductive mounting) and the display of aggression following male-directed mounting (aggressive mounting). Using optogenetics, we further identified genetically defined neural populations in the medial preoptic area (MPOA) that mediate reproductive mounting and the ventrolateral ventromedial hypothalamus (VMHvl) that mediate aggressive mounting. In vivo microendocsopic imaging in MPOA and VMHvl revealed distinct neural ensembles that mainly encode either a reproductive or an aggressive state during which male or female directed mounting occurs. Together, these findings demonstrate that internal states are represented in the hypothalamus and that motorically similar behaviors exhibited under different contexts may reflect distinct internal states.