Humans, dolphins, and elephants are the only known species that vocally label their conspecifics. It remains unclear whether nonhuman primates share this ability. By recording hundreds of spontaneous “phee-call” dialogues between pairs of marmoset monkeys we discovered that marmosets use these calls to vocally label their conspecifics. Moreover, by using closed-loop playback experiments we found that they respond more consistently and correctly to calls that are specifically directed at them. Analysis of calls from multiple monkeys revealed that family members use similar calls and acoustic features to label others and perform vocal learning. These findings reveal unexpected vocal flexibility and vocal learning among nonhuman primates and suggest that marmosets’ vocal labeling of others and its neuronal circuits may provide new insights into the evolution of human speech.