Abstract structure and generalization in sensorimotor networks configured with semantic-based instruction embeddings
Reidar Riveland & Alex Pouget · COSYNE 2023
One of the most essential language skills that humans possess is the ability to correctly execute actions based on linguistic instructions. Here we use the latest advances in Natural Language Processing (NLP) to create a neural model of this cognitive ability. Models are trained on a set of 50 psychophysical tasks which require cognitive competencies well studied in the literature. Information about tasks demands is embedded using a transformer architecture pre-trained on an NLP task. Our best performing model, clipNet, uses 33M parameters pre-trained on embedding images and their correspondin
Sun, Mar 12, 2023