When making decisions in a cluttered world, humans and other animals often have to hold multiple items in memory at once -- for instance, remembering the different items on a shopping list. Interestingly, psychophysical experiments in humans and other animals have shown that sometimes the remembered stimuli become confused; that is, instead of reporting a stimulus that was actually present, the participant will occasionally report a chimeric stimulus, composed of feature values that were present but bound together in combinations that were not (e.g., setting out to buy a green apple and red pear, but ending up with a red apple and green pear). These errors are referred to as swap, or misbinding, errors -- and while they have been described behaviorally, their neural mechanisms are, to our knowledge, unknown. We show how these swap errors manifest in neural activity from both posterior and frontal brain regions while monkeys perform a multi-stimulus working memory task. In the task, the monkeys must remember the color and location of two stimuli. After a delay, a learned cue indicates which location held the target color. After a second delay, the animal reports the target color on a continuous color wheel. We use a behavioral model to infer the likelihood that each response is a swap error, and combine this with a model of the neural activity that reflects different hypotheses about how representations change during swap errors relative to correct responses. Surprisingly, we find that swap errors are associated with a misbinding of color to location during the selection process, rather than with a misinterpretation of the cue, or an error encoding the stimuli. More broadly, these results suggest that selection from working memory may be key source of catastrophic behavioral errors, indicating a potentially fruitful focus for future research.