Continual learning (CL), the capability to acquire knowledge and skills over time without forgetting old ones, is fundamental to how animals survive in a non-stationary world. This ability has proven difficult to realize in artificial neural networks (ANNs) [1], which suffer from catastrophic forgetting of previously learned information. To understand this contrast, we sought to understand what aspects of a neural system and the tasks it learns are key to its CL performance. We developed a theory of deep, nonlinear feedforward networks sequentially learning multiple tasks and derived exact expressions of how the network’s input output function changes over the course of learning. Subsequent analysis identifies two order parameters (OPs) that measure the relations between different tasks in the network’s representation space and are highly predictive of forgetting severity. They are defined in terms of the subspaces spanned by representations of different tasks’ inputs as well as “rule vectors” that encode the learned input-output rules in the tasks. Larger projections of rule vectors onto the subspace shared by tasks or greater dissimilarity between rule vectors result in faster forgetting. We validated their predictions using a novel synthetic task-sequence paradigm, where we can parametrically vary task relations, as well as a variety of task sequences of benchmark datasets. Thus, they may capture how the geometry of representations of different tasks affects CL performance in ANNs and guide further experimental investigations of neural basis of CL.