Correlated activity between neurons can cause variability in behavior across trials, as trial-by-trial neuronal noise will propagate through the motor system. The extent to which correlated activity affects behavior depends on the properties of its translation into movement. A major hurdle in studying the effect of neural correlations on behavior is the need to assume a model of the relationship between neural activity and behavior. We developed a novel method that estimates the contribution of correlations to behavior, with minimal assumptions. We applied this method to the frontal eye field (FEF) and pursuit eye movements behavior. We defined a distance metric between the behavior on different trials. Based on this metric, we applied a sequence of shuffles to the neuronal responses, allowing trials to be matched with increasingly distant trials. Although correlations were partially explained by behavior, even the most constrained shuffle strongly attenuated the correlations. Thus, only a small fraction of FEF correlations affect the behavior. We used simulations to validate our approach and show that it captures the correlations that affect behavior. The simulations also demonstrate the generalizability of our method over different models. We show that the attenuation of correlated activity through the motor pathway could stem from the interplay between the structure of the correlations and the decoder of FEF activity.