Latent variable models are widely used in neuroscience to extract dynamical structure from high-dimensional neural activity, and provide valuable insights into neural computation. However, these models are typically inferred from a single recording session and exhibit limited generalization capabilities. This is primarily due to statistical heterogeneities in neural recordings across subjects and tasks, along with lack of inductive biases for disentangling shared and dataset-specific dynamical structure. In this work, we hypothesize that similar behavioral tasks admit a corresponding family of related solutions and propose a novel framework for meta-learning this solution space from task-related neural activity of trained animals. Specifically, our approach captures the variations in the latent dynamical structure across recordings in a low-dimensional vector, which we refer to as the dynamical embedding. During training, the model learns to adapt a hierarchical latent dynamical system model conditioned on the dynamical embedding via low-rank changes to the model parameters. We test our approach on synthetic dynamical systems and population recordings from the motor cortex in monkeys during different reaching tasks. We demonstrate that our method is able to successfully capture diverse dynamical behaviors, and the manifold spanned by the embedding vectors offers a useful inductive bias for learning latent dynamics from limited samples of novel recordings. To the best of our knowledge, this is the first approach that facilitates learning a family of dynamical systems from diverse recordings in a unified latent space while providing a concise, interpretable manifold over dynamical variations. By leveraging shared structure across related tasks, our approach significantly enhances our ability to integrate, and analyze complex neural dynamics across different experimental conditions, broadening the scope of scientific inquiries possible in neuroscience and paving the way for more robust and generalizable models of neural dynamics.