Computational Linguistics seminars

This talk surveys the components of modern diffusion language models, beginning with masked prediction and extending to iterative refinement, controlled generation and variable-length inference. It examines how parallel generation and repeated revision change inference-time computation. Examples from natural language, proteins and DNA illustrate both practical model designs and uses of fine-grained control in scientific sequence generation.

Large speech and language models offer testable algorithmic hypotheses about how the brain understands language. Laura Gwilliams examines neural recordings at several spatial scales, from population activity measured with MEG and intracranial EEG to individual neurons sampled across cortical depth with human Neuropixels probes. The studies ask which representations and operations connect speech sounds to meaning, and how information at different timescales is nested across time and cortical space. Comparing biological recordings with artificial models reveals candidate mechanisms for exchanging information across hierarchical structures and for implementing language understanding at the level of whole neural systems.

Small language models trained on limited data can provide economical, controlled simulations for studying human language acquisition. Alex Warstadt discusses the BabyLM Challenge and its goals of improving data efficiency, cognitive plausibility and accessibility. Studies from his UC San Diego lab train models from scratch under controlled conditions to investigate critical periods in second-language learning, inductive biases behind cross-linguistic patterns, and distributional signatures of word-learning trajectories. Comparing cases where these models resemble human learners with cases where they diverge helps locate human language acquisition within a broader space of possible learning systems. The approach supports fundamental work across machine learning, linguistics and cognitive science.

What can fluent language models tell us about linguistic structure and human language learning? Kyle Mahowald evaluates two opposing positions: that models do not learn language in a way relevant to people, and that their success makes linguistic theory unnecessary. Drawing on a position paper with Richard Futrell, he argues that language models provide useful evidence about structure, processing and acquisition while leaving an essential role for theory. The talk examines grammaticality judgments in language models and controlled pretraining experiments in which small models learn from systematically manipulated corpora. These methods allow researchers to revisit claims about learning and identify which properties of the input support linguistic generalisation.

Modern artificial intelligence (AI) systems are powered by foundation models. This paper presents a new set of foundation models, called Llama 3. It is a herd of language models that natively support multilinguality, coding, reasoning, and tool usage. Our largest model is a dense Transformer with 405B parameters and a context window of up to 128K tokens. This paper presents an extensive empirical evaluation of Llama 3. We find that Llama 3 delivers comparable quality to leading language models such as GPT-4 on a plethora of tasks. We publicly release Llama 3, including pre-trained and post-trained versions of the 405B parameter language model and our Llama Guard 3 model for input and output safety. The paper also presents the results of experiments in which we integrate image, video, and speech capabilities into Llama 3 via a compositional approach. We observe this approach performs competitively with the state-of-the-art on image, video, and speech recognition tasks. The resulting models are not yet being broadly released as they are still under development.

Natural language understanding comprises a wide range of diverse tasks such as textual entailment, question answering, semantic similarity assessment, and document classification. Although large unlabeled text corpora are abundant, labeled data for learning these specific tasks is scarce, making it challenging for discriminatively trained models to perform adequately. We demonstrate that large gains on these tasks can be realized by generative pre-training of a language model on a diverse corpus of unlabeled text, followed by discriminative fine-tuning on each specific task. In contrast to previous approaches, we make use of task-aware input transformations during fine-tuning to achieve effective transfer while requiring minimal changes to the model architecture. We demonstrate the effectiveness of our approach on a wide range of benchmarks for natural language understanding. Our general task-agnostic model outperforms discriminatively trained models that use architectures specifically crafted for each task, significantly improving upon the state of the art in 9 out of the 12 tasks studied. For instance, we achieve absolute improvements of 8.9% on commonsense reasoning (Stories Cloze Test), 5.7% on question answering (RACE), and 1.5% on textual entailment (MultiNLI).