Recent studies have demonstrated that brain areas involved in motor skills display common representations of latent trajectories for both stereotypical and generalizable dynamical cortical behavior. However, current research often focuses on analyzing single tasks and individual sessions, missing the opportunity to exploit the structural patterns across datasets. By fitting a unified model to a collection of neural data with shared features, we can increase data usage efficiency and enhance our understanding of common neural computation structures, despite the challenge of distinct populations of neurons in each recording. In this work, we present a new approach we call the Multi-X Denoising Diffusion Model (Multi-X DDM), which circumvents the need for explicit alignment between multiple sessions by using the implicit alignment capabilities of diffusion models. The quality of the learned dynamics is evaluated through the model's forecasting ability, which predicts multiple time-steps for both neural activity and behavior. Our results show that the pretrained model can be efficiently adapted to novel, unseen sessions without requiring explicit neuron correspondence. This enables few-shot learning with minimal labeled data, as well as competitive performance in zero-shot learning. Moreover session embeddings cluster based on subjects' strategies, and the model reveals neurons that are important for behavior. In addition, we introduce NeuroTask, a benchmark dataset with six electrophysiology datasets, seven behavioral tasks, 19 animals, and 261 sessions, and an API for efficient data handling.