Cortical neuronal networks exhibit a rich dynamical repertoire and are capable of performing many different computations, for instance the generation of multiple distinct dynamical patterns in motor cortex. However, we do not know what connectivity structures support this capability, nor the mechanism for selecting among the dynamics required for different tasks. We develop an analytical theory describing a network model capable of multi-task computation, where the connectivity is generated as a sum over a large number of low-rank matrices, each of which is specialized to perform one low-dimensional task. Such networks exhibit a "spontaneous" state in which no particular task is active, and the network implements a linearized, noise-corrupted version of each task's dynamics simultaneously. In this state, neuronal gains and noise levels are stationary, self-consistently determined by averages over all tasks. Task selection occurs via a phase transition between this spontaneous state and a state in which the effective connectivity corresponding to the selected task is amplified and loses linear stability. Nonlinear dynamics in the task-relevant subspace then dominate the network dynamics. This model provides several insights into the properties of networks that can multi-task. First, it demonstrates that an extensive number of tasks can be embedded in the dynamics of a single network, with each task occupying a different subspace of neural activity. Such an organization is consistent with large-scale recordings of motor areas during the generation of distinct behaviors. Second, it suggests that time-varying spontaneous activity may be a consequence of learning to perform many different tasks. Finally, it provides a new and analytically tractable description of the processes behind task selection. We discuss how this selection process may be implemented either through inputs to the network or through modulation of its effective connectivity.