Multi task representations of RNNs show a simplicity bias

Recently, both experimental and theoretical work has expanded from studying neural systems with one task at a time to studying them while performing multiple tasks. An open question is understanding how a brain area, a population of inter-connected neurons, can carry on different computations depending on the context. Computations can either share the neural population code, share some activity modes, be entirely separated in activity space, or anything in between. To study this theoretically, it is possible to train RNNs on multiple tasks and study the emerging hidden representations. The challenge is that tasks that are typically used in Neuroscience are composed of multiple sensory, cognitive, and motor primitives, and traditional task-similarity is multidimensional, which makes it difficult to know which computational component influenced the final representations. To overcome that, we develop a framework that disambiguates cognitive computation from sensory and motor processes, by designing all tasks to operate on identical input and output statistics. The framework naturally leads to simple mathematical measures of task-similarity. We train multiple RNN instances on a large set of tasks of two types (1-back and 2-back memory), with various architectural choices. We find that networks are \textit{simplicity-biased}, in the sense that they will form the minimal dynamical objects necessary to solve the task. We demonstrate the interaction between this bias and the external forces -- task-similarity, input and output architecture, and context signals -- shape the joint representation.