The surge in interest in individual differences, driven partly by the rise of precision medicine, has coincided with the latest replication crisis, centred around brain-wide association studies of brain-behaviour correlations. In this context, understanding the reliability of the measures we use in cognitive neuroscience has never been more important. Tasks designed to study individual differences must reliably differentiate participants at the individual rather than the group level. Yet assessing the reliability of a given dataset, and even more so optimizing the design of a new behavioural task to ensure sufficient reliability, is a complex problem. Here, we first show that behaviour at the individual level does indeed converge to a stable mean, given sufficient data, allowing reliable separation of individuals. We next mathematically derive a simple function that describes the reliability of behavioural measures using basic statistics of the distribution. We tested this function extensively on a large real-world dataset collected on a battery of commonly used tasks across different cognitive domains and supplemented by large-scale simulations of synthetic datasets. We subsequently used this function to develop a simple web-based online tool that can predict the expected reliability for any given number of trials and participants, even based on just initial pilot data. Lastly, we examine the effect on the reliability of measuring over different time points. We show that tasks assessing different cognitive domains are differentially affected, with those involving memory or attention most affected. Averaging over 2-3 sessions appears to restore the trait-like stability of even affected measures.