Quantifying the contribution of dynamically changing neural activity to goal-directed behaviors is a longstanding problem in neuroscience. One approach is to systematically test how behaviors map to individual firing rates at each time step, but it is less clear how behaviors map to complex neural latent dynamics. In recent years deep learning approaches have become powerful tools to extract these latent dynamics (Schneider, Lee, Mathis 2023), yet how to probe them in order to attribute input neurons to latents remains challenging. Gradient-based attribution is one of the methods that aim to explain decisions of deep learning models, but so far lack identifiability guarantees. Here, we propose a method to generate attribution maps with identifiability guarantees by developing a regularized contrastive learning algorithm trained on neural data with continuous behavioral labels. We empirically verify the theoretical guarantees on synthetic data that simulates the spatial modulation of cells in the entorhinal cortex (George et al., 2023). Moving beyond synthetic data, our model is able to correctly identify grid cells from freely-moving rodents (Gardner et al., 2021), outperforming or matching previous attribution methods based on feature ablation, Shapley values and other gradient-based methods, yet with less computational cost. Importantly, this approach can be employed as a viable alternative to classifying cells based on traditional scores (e.g., grid scores), thereby providing a less biased and more generalizable approach to measuring the neural contribution of behavior.