Is It Time to Rethink LLM Pre-Training?

Aditi Raghunathan examines limits of next-token prediction for creative and adaptable language models. The discussion covers structured randomness, alternative training objectives, catastrophic overtraining and the gap between benchmark scores and fine-tuning flexibility. It also explores memorisation sinks for targeted unlearning and greater control over models.