Do neural networks understand what they are doing?

Neural networks are often said to build models of the world ("world models") rather than merely exploit heuristics, and in that sense to understand their tasks, yet it is unclear what having a world model amounts to or what evidence should count for or against one. The first half of this talk proposes a minimal notion of a world model, a single unified representation of a task that preserves its structural relations, places it within a hierarchy of stronger notions, and argues that many disputes about world models are merely verbal. The second half re-examines the standard example of a network using a world model, Othello-GPT, a transformer widely taken to model the Othello board, arguing that the existing evidence is ambiguous once appropriate baselines and metrics are used. It then introduces a further kind of evidence favouring the heuristic interpretation: how quickly the network learns Othello variants that extend, rather than scramble, the geometry of the board. The talk closes with general lessons for attributing world models to neural networks and with similar approaches for attributing algorithms and probabilistic inference.