Our ability to form and retrieve memories relies on a network of connected brain regions. In particular, neuronal ensembles across regions form distributed memory engrams that have structural and functional connectivity. However, the computational principles underlying the distributed nature of memory remain unclear. Here, we show that distributed engrams enable memory generalization and discrimination in parallel across brain regions. Orthogonal and complementary, generalization and discrimination are computations that must be balanced for adaptive, memory-guided behavior. For instance, while animals need to generalize threat-predictive cues to novel stimuli with shared features, they also need to discriminate between threat-predictive and neutral cues. To investigate the dichotomy between memory generalization and discrimination, we modeled multi-region spiking neural networks with state-dependent and region-specific synaptic plasticity. Our model predicted that while distributed engrams exhibit representational drift, neural dynamics retain a stable geometry: neuronal activity is either scaled linearly or rotated orthogonally over time. Our model also predicted that distributed engrams generalize and discriminate in parallel by bringing the representations of training and novel stimuli either close together or far apart, respectively. Surprisingly, our model proposed that generalizing and discriminating regions engage in adversarial representation learning. Using in vivo longitudinal calcium imaging in mouse thalamus, hippocampus, and cortex, we performed associative learning experiments that supported our model’s predictions. Our results revealed that the distributed organization of memory enables parallel orthogonal computations --- a potentially general computational principle of distributed neural representations.