Recent studies have proposed that neural circuits generate a task manifold: i.e., a subset of the neural state-space to which neural activity is confined as an animal performs a task. Thus, discovering and characterising these manifolds from experimental data can shed light on the neural computations unfolding within the brain during behavioural tasks. In parallel, neuroscience has seen a recent surge of interest into fitting dynamical systems to experimental data with the goal of reverse-engineering neural circuits. Yet, common neural manifolds frameworks often do not take into account that neural activity is generated by an underlying dynamical system. Here, we argue that viewing neural manifolds as mathematical manifolds through the lens of differential geometry provides a natural link between the geometric and dynamical systems perspectives on neural activity. Building on these results, we introduce a data-driven framework to model the geometry and dynamics of neural population activity: Manifold Discovery via Dynamical Systems (MDDS). In recordings of macaque motor and premotor cortex during a reach task, we show that MDDS uncovers a manifold with behaviourally-relevant geometry. Neural trajectories on that manifold closely resemble the hand movement of the animal, without a need to explicitly decode the behaviour. Furthermore, from 2-photon imaging of mouse visual cortex, we show that neurons tracked over one month of learning can have a stable curved manifold shape, while the dynamics on this manifold change. Overall, our framework offers a formal mathematical link between the geometric and dynamical perspectives on population activity, and provides a generative model to uncover task manifolds from experimental data. We use this framework to highlight how behavioural variables are naturally encoded on curved neural manifolds, and how this encoding evolves over learning.