Abstract structure and generalization in sensorimotor networks configured with semantic-based instruction embeddings

One of the most essential language skills that humans possess is the ability to correctly execute actions based on linguistic instructions. Here we use the latest advances in Natural Language Processing (NLP) to create a neural model of this cognitive ability. Models are trained on a set of 50 psychophysical tasks which require cognitive competencies well studied in the literature. Information about tasks demands is embedded using a transformer architecture pre-trained on an NLP task. Our best performing model, clipNet, uses 33M parameters pre-trained on embedding images and their corresponding text captions. The resultant instruction embeddings allowed the network to perform a previously unseen tasks at 85% correct on average based exclusively on information in the instruction (i.e. zero-shot learning). Surprisingly, gptNetXL, which is pre-trained on predicting the next word in a sentence and uses 1.5 billions parameter to embed instructions, exhibits a generalization performance of just 68%. In clipNet we found that language scaffolds sensorimotor representations such that activity for interrelated tasks share a common geometry with the semantic representations of instructions, allowing language to cue the proper composition of practiced skills in unseen settings. This structure was largely absent in gptNetXL. We show that this geometry is present throughout the language processing hierarchy for clipNet but not gptNetXL. Finally, we also demonstrate how clipNet can generate a linguistic description of a novel task it has identified using motor feedback, which can subsequently guide a partner model to perform the task. Our models offer experimentally testable predictions outlining how linguistic information must be represented in order to enable flexible cognition.