In neuroscience, we record neural activity over time. We can try to make sense of this data by studying the (low-dimensional) space the activity is constrained to. To identify these spaces, one typically either treats the rows (neurons / spike trains) or the columns (timepoints / population vectors) of the activity matrix as points. In general, these two perspectives will not give the same space, obscuring the underlying structure of the data. Here, we provide a unified framework for thinking about this challenge in neural data. We show that if rows and columns are convex on the space(s), we recover the same space in both. This is already the case in a population of head direction cells, where both rows and columns encode the head direction circle. Here, neurons (rows) are convex as they have clear tuning to a single direction and timepoints (columns) are convex as the head can only point in one direction at a time. Unfortunately, neural recordings are often more complex. For instance, if we record from several distinct spaces simultaneously, such as the head direction circle and the grid cell torus, the columns will be non-convex, resulting in the product space. Similarly, we get non-convex rows by recording from neurons that have more than one firing field, either on the same space or on different spaces. In this case, we need to transform the data into components that are all convex on a single space. If the data comes from several spaces, this disentangles the components and avoids the curse of dimensionality. We show that an adapted Non-negative Matrix Factorization (NMF; [Lee,1999]) can do this. Furthermore, we give reconstruction results, showing that this indeed gives the true space in both rows and columns and give examples from simulated and recorded data.