Next-token prediction

Andrew Dai joins The Information Bottleneck to discuss language-model pre-training, next-token prediction, training-data quality and visual reasoning. The conversation examines counting and image understanding, world models and JEPA. Hosted by Ravid Shwartz-Ziv and Allen Roush.

…training experience, she discusses next-token prediction, attention and loss functions, trajectory data, context retrieval, error recovery and the role of scale in learning tool use.

Aditi Raghunathan examines limits of next-token prediction for creative and adaptable language models. The discussion covers structured randomness, alternative training objectives, catastrophic overtraining and the gap between benchmark scores and fine-tuning flexibility. It also explores memorisation sinks for targeted unlearning and greater control over models.