Topic: linguistic tasks

LSTMs trained on next word prediction can accurately perform linguistic tasks that require tracking long-distance syntactic dependencies. Notably, model accuracy approaches human performance on subject-verb number agreement tasks including cases with interfering attractors (Gulordava et al., 2018). However, we do not have a mechanistic understanding of how LSTMs track syntactic structures to perform such linguistic tasks. Do LSTM language models learn abstract grammatical rules like humans, or do they rely on simple heuristics and patterns? Here, we test this using long-distance gender agreeme