While several methods exist for clustering neurons based on their selectivity to specific visual features, the extent to which neurons form clusters based on their invariances - how neurons maintain consistent responses despite variations in visual input - remains largely unexplored. Clustering neurons based on their invariances, however, presents several challenges, including the identification of directions in the stimulus space along which a neuron exhibits invariance (i.e., consistent response), and the comparison and alignment of such non-trivial trajectories to assess whether neurons exhibit similar invariances. Here, we introduce a novel approach, which builds upon existing method using Implicit Neural Representations and predictive models of neural responses to effectively capture the non-trivial invariance manifold of maximally exciting stimuli for single neurons [1]. Once the invariance manifold of a given neuron is captured, our method learns an affine transformation on the pixel coordinates such that the same manifold maximally activates another neuron. Neurons that their manifolds highly activate each other after the corresponding affine transformation are then grouped into the same cluster. When applied to responses recorded from V1 neurons, our method yields multiple functional clusters exhibiting invariances beyond the commonly expected phase-invariance. Overall, our proposed method can be a valuable tool for a systematic and quantifiable exploration of neural invariances landscape, enabling functional classification of visual sensory neurons based on their invariances.1] Baroni, Luca and Bashiri, Mohammad et al. "Learning invariance manifolds of visual sensory neurons." NeurIPS Workshop on Symmetry and Geometry in Neural Representations. PMLR, 2023