In vivo, single neurons have high trial-to-trial variability in their information transmission, which presents a challenge for networks whose outputs must be consistent for high task performance. Reliability becomes increasingly difficult to attain for working memory tasks where information must be preserved for extended periods of time. This implies that relative to systems without noise, the brain must have mechanisms dedicated to compensating for unreliability in neuron activity. How trained networks mitigate the effects of noise while completing working memory tasks remains largely unexplored, primarily due to the lack of a theoretical framework for quantifying and analyzing noise compensation in neural networks. We take a step in this direction by analyzing noisy recurrent neural networks (RNNs) trained to perform a delayed replication task, where noisy neurons hold inputs in memory for several time steps before outputting the same signal. Noise makes this task increasingly difficult, because information becomes progressively corrupted throughout time. We develop a principled method to quantify noise compensation across temporal trajectories, and show that trained networks reduce noise within a low-dimensional `mechanistic' activity space by maximizing the signal-to-noise ratio for highly probable inputs, and quenching low probability inputs, which are likely due to noise. Further, we show that this compensation phenomenon can be understood in terms of the implicit regularization introduced by training a system under noise, as it does not exist in networks that are trained without noise. From this simple example, our analysis suggests a more general framework for exploring the noise compensation properties of neural networks engaged in working memory tasks, which require holding information in memory for extended periods of time.