‘Surprise’ is used in the neurosciences to explain a multitude of phenomena: from memory-modification and adaptive learning to attentional shift and exploration. However, different studies refer to different definitions of ‘surprise’, and it remains unclear how these different definitions relate (1) to each other and (2) to different phenomena attributed to surprise. To address the 1st issue, we identify 10 mathematical definitions of surprise in a unifying framework and show how they relate to each other. Importantly, we classify these definitions into four main categories: (i) ‘prediction surprise’ measures a mismatch between a prediction and an observation; (ii) ‘change-point detection surprise’ measures the probability of a change in the environment; (iii) ‘confidence-corrected surprise’ explicitly accounts for the effect of confidence; and (iv) ‘information gain surprise’ measures the belief-update upon a new observation. To address the 2nd issue, we focus on surprise-modulated adaptive learning and surprise-seeking exploration. We propose two experimental paradigms where different categories (i-iv) make different predictions: an association learning task in volatile environments and a model-building task in reward-free bandits. In these two paradigms, modeling shows that surprise-modulation of the speed of learning leads to sensible adaptive behavior only for change-point detection surprise whereas surprise-seeking leads to sensible exploration strategies only for information gain surprise. We, therefore, conclude that there cannot be a single surprise measure with all functions and properties previously attributed to surprise. Overall, our results unify multiple theories of surprise and propose a theory-driven approach to design experiments on the physiological signatures and functional roles of surprise in the brain.