Natural Language Processing seminars

Deep networks display recurring phenomena such as grokking, scaling behaviour, generalization despite overparameterization and failures outside training distributions. This talk surveys controlled settings where those behaviours can be explained, suggesting that language models may combine many tractable, largely interpolative mechanisms. It then distinguishes that scientific account from a more speculative argument about the role of industrial computation and human labour, and what these imply for claims of broadly extrapolative intelligence.

This talk surveys the components of modern diffusion language models, beginning with masked prediction and extending to iterative refinement, controlled generation and variable-length inference. It examines how parallel generation and repeated revision change inference-time computation. Examples from natural language, proteins and DNA illustrate both practical model designs and uses of fine-grained control in scientific sequence generation.

Masked diffusion language models can reveal tokens out of order and in parallel, potentially reducing sequential inference costs. This talk asks how much parallelism is possible before distributional accuracy degrades. It gives an exact expression for expected divergence under arbitrary data distributions and unmasking schedules, exposing a connection with classical approximation of one-variable functions.

Three projects investigate world modelling and state computation in frontier AI systems. Two evaluate dual-perspective reasoning and knot manipulation. The third tests the state-separation hypothesis: that separating the mechanism for predicting outputs from the mechanism for computing state can improve language-model performance.

Behaviours that resemble world knowledge do not by themselves reveal what an AI represents internally. This talk uses Craik's account of mental models—supporting alternatives, prediction and transfer—to frame empirical tests. Spatial reasoning in language models and motion forecasting in visual models illustrate ways to probe those abilities, along with the limits of inferring world models from systems trained on human-produced representations.

Adam Brown reviews recent progress in training language models to perform scientific work and reasoning. The talk considers how the rapid growth of these capabilities could change theoretical-physics research, while distinguishing present achievements from speculation about the consequences of continuing progress.