In animals and humans, curriculum learning---presenting data in a curated order---is critical to rapid learning and effective pedagogy. A long history of experiments has demonstrated the impact of curricula in a variety of animals---including rats, mice, dogs, pigeons and humans---but, despite its ubiquitous presence, a theoretical understanding of the phenomenon is still lacking. Surprisingly, in contrast to animal learning, curricula strategies are rarely used in machine learning and recent simulation studies conclude that curricula are moderately effective or ineffective in most cases. This stark difference raises a fundamental theoretical question: when and why does curriculum learning help? In this work, we analyse a prototypical model of curriculum learning in the high-dimensional limit, employing statistical physics methods. We study a task in which a set of informative features are embedded amidst a large set of noisy features. We analytically derive average learning trajectories for simple neural networks on this task, which establish a clear speed benefit for curriculum learning in the online setting. However, when training experiences can be stored and replayed (for instance, during sleep), the advantage of curriculum in standard neural networks disappears, in line with observations from ``batch'' training in the deep learning literature. Inspired by synaptic consolidation techniques developed to combat catastrophic forgetting, we investigate whether consolidating synapses at curriculum change points can boost the benefits of curricula. We derive generalization performance as a function of consolidation strength (implemented as Gaussian priors connecting learning phases), and show that this consolidation mechanism can yield large improvements in test performance. Taken together, our analytical descriptions help reconcile apparently conflicting empirical results, trace regimes where curriculum learning yields the largest gains, and provide experimentally-accessible predictions for the impact of task parameters on curriculum benefits. More broadly, our results suggest that fully exploiting a curriculum may require explicit consolidation at curriculum boundaries.