The Information Bottleneck

Yuandong Tian discusses his research on computer Go, from DarkForest to OpenGo, and the role of action-space design in applying reinforcement learning. The conversation covers gradient-free optimization, neural architecture search, Coconut’s approach to reasoning in latent space, representation learning and grokking. It then examines AI-assisted research, recursive self-improvement, coding-agent limitations, and whether alternative architectures can outperform transformers. The closing discussion considers data efficiency, robotics, restrictions on self-improving systems, and open-source models.

Hosted by Ravid Shwartz Ziv, Allen Roush

Stanford linguist and computer scientist Chris Manning discusses what linguistics contributed to machine learning and why pragmatics and dialogue remain open problems for language models. The conversation examines early abstraction of verb categories in small transformers, how distributed representations shape learning, and whether language alone can support meaningful representations. It also considers language in world models, diffusion language models, and representation finetuning (ReFT), which steers frozen models through their hidden states. The final discussion asks where knowledge resides and whether concepts occupy linear subspaces.

Hosted by Ravid Shwartz Ziv, Allen Roush

Andrew Dai joins The Information Bottleneck to discuss language-model pre-training, next-token prediction, training-data quality and visual reasoning. The conversation examines counting and image understanding, world models and JEPA. Hosted by Ravid Shwartz-Ziv and Allen Roush.

Carnegie Mellon researcher and Abridge co-founder Zachary Lipton discusses where machine learning can help healthcare and why clinical work is harder to automate than software development. The conversation examines medical documentation, decision support, drug discovery, open models and model routing, then turns to academic research, doctoral training and the effects of automation on scientific careers. Hosted by Ravid Shwartz Ziv and Allen Roush. Watch the full conversation on the publisher’s YouTube channel.

Hosted by Ravid Shwartz Ziv, Allen Roush

Sara Hooker of Adaptation Lab explores AI systems that change with their users, tasks and environments. The discussion connects continual learning, efficient adaptation and automated science with the difficulty of evaluating tasks that lack easily checked answers. It also covers interfaces, distillation, open models, safety and regulation, multilingual tokenization, and possible successors to current Transformer systems. Hosted by Ravid Shwartz Ziv and Allen Roush. Watch the full conversation on the publisher’s YouTube channel.

Hosted by Ravid Shwartz Ziv, Allen Roush

Microsoft researcher Alexia Jolicoeur-Martineau explains the Tiny Recursive Model and the reasoning behind her work on small networks that repeatedly refine an internal state and proposed answer. The conversation examines truncated gradients, puzzle-solving benchmarks and the differences from autoregressive language generation. It also considers molecular modelling, limits of data scaling and which research directions she would prioritize. Hosted by Ravid Shwartz Ziv and Allen Roush. Watch the full conversation on the publisher’s YouTube channel.

Hosted by Ravid Shwartz Ziv, Allen Roush

Rohan Anil of Core Automation discusses the relationship between pretraining, reinforcement learning and learning after deployment. Drawing on his optimization work at Google and Anthropic, he examines why additional post-training, distillation and longer contexts may not solve continual learning. Other topics include second-order optimizers, distributed training, low-level coding agents and the difficulty of coordinating very large GPU clusters. Hosted by Ravid Shwartz Ziv and Allen Roush. Watch the full conversation on the publisher’s YouTube channel.

Hosted by Ravid Shwartz Ziv, Allen Roush

John Langford of Microsoft AI Labs discusses whether compact implicit models of the world can improve data efficiency. The conversation examines belief-state compression, Transformer caches, JEPA-style objectives and his Next Latent research. It also covers the value of algorithmic research alongside scaling, agent-assisted experiments, open models, the history of CAPTCHA and the behavior of modern optimizers. Hosted by Ravid Shwartz Ziv and Allen Roush. Watch the full conversation on the publisher’s YouTube channel.

Hosted by Ravid Shwartz Ziv, Allen Roush

INRIA researcher David Holzmüller explains how to choose among tabular foundation models, boosted trees and multilayer perceptrons. He discusses TabArena, TabICL and RealMLP, benchmark weaknesses, small-data performance, calibration and class imbalance. The conversation also examines text columns, explainability, time-series distinctions, synthetic pretraining and the trade-offs among computation, inference speed and predictive performance. Hosted by Ravid Shwartz Ziv and Allen Roush. Watch the full conversation on the publisher’s YouTube channel.

Hosted by Ravid Shwartz Ziv, Allen Roush

Charles Frye of Modal examines the infrastructure constraints behind large-scale AI research. With Ravid Shwartz Ziv and Allen Roush, he discusses underused hardware, saturated resources, sharing GPU capacity and deciding when to train a model. The conversation also covers inference and speculative decoding, hardware economics, virtualization and the security boundaries around agents and open models. Hosted by Ravid Shwartz Ziv and Allen Roush. Watch the full conversation on the publisher’s YouTube channel.

Hosted by Ravid Shwartz Ziv, Allen Roush

EleutherAI executive director Stella Biderman discusses open models, independent research and the containment of potentially dangerous AI capabilities. She argues for stronger isolation in security evaluations and examines the influence of corporate power on scientific evidence. Other topics include sovereign AI, international access to open models, the role of interfaces in perceived progress, and filtering hazardous knowledge during pretraining. Hosted by Ravid Shwartz Ziv and Allen Roush. Watch the full conversation on the publisher’s YouTube channel.

Hosted by Ravid Shwartz Ziv, Allen Roush

University of Freiburg professor and Prior Labs co-founder Frank Hutter explains how TabPFN changed deep learning for tabular data. The episode examines in-context learning from synthetic pretraining, forecasting, architectural approaches to larger tables and the continuing roles of boosted trees and language models. Hutter also discusses moving research into a company, team autonomy, model licensing and building research organizations outside established technology hubs. Hosted by Ravid Shwartz Ziv and Allen Roush. Watch the full conversation on the publisher’s YouTube channel.

Hosted by Ravid Shwartz Ziv, Allen Roush

Stanford researcher Surya Ganguli connects statistical physics, theoretical neuroscience and machine learning. He discusses scaling laws, data selection and the origins of diffusion models, then turns to neural experiments on perception, self-related processing and describing neuronal responses. The conversation closes with questions about human versus machine data efficiency and the different ways brains and artificial networks acquire useful algorithms. Hosted by Ravid Shwartz Ziv and Allen Roush. Watch the full conversation on the publisher’s YouTube channel.

Hosted by Ravid Shwartz Ziv, Allen Roush

Google DeepMind research director Brendan O'Donoghue examines discrete diffusion as an alternative to autoregressive text generation. He discusses sample diversity, reinforcement learning, latency and potential on-device or robotics applications. The episode also addresses the costs of serving diffusion models, hardware trade-offs between computation and memory bandwidth, and practical limits on progress toward more capable AI. Hosted by Ravid Shwartz Ziv and Allen Roush. Watch the full conversation on the publisher’s YouTube channel.

Hosted by Ravid Shwartz Ziv, Allen Roush

Nathan Lambert discusses post-training and the prospects for open AI models, drawing on his work on OLMo at Ai2 and his writing on reinforcement learning from human feedback. Topics include capability gaps, the economics of the open ecosystem, training environments, continual learning and skepticism about recursive self-improvement. The conversation also examines research culture, concentrated talent and whether increased model spending translates into better products. Hosted by Ravid Shwartz Ziv and Allen Roush. Watch the full conversation on the publisher’s YouTube channel.

Hosted by Ravid Shwartz Ziv, Allen Roush

Daphne Koller of insitro discusses the differences between finding predictive patterns and identifying interventions that improve disease outcomes. The conversation examines biological data scarcity, structure and causality, failures in drug development and the limits of speeding up wet-lab work with AI agents. It also considers foundation models for biology, generalization beyond observed data, human evidence and scientific discovery. Hosted by Ravid Shwartz Ziv and Allen Roush. Watch the full conversation on the publisher’s YouTube channel.

Hosted by Ravid Shwartz Ziv, Allen Roush

PufferAI researcher Joseph Suarez argues that implementation and simulation bottlenecks have constrained reinforcement learning as much as algorithm choice. He discusses the design of useful simulators, CPU and GPU trade-offs, faster training pipelines and open-source research tools. The conversation ranges from game environments to potential applications in scientific simulation and biological modelling. Hosted by Ravid Shwartz Ziv and Allen Roush. Watch the full conversation on the publisher’s YouTube channel.

Hosted by Ravid Shwartz Ziv, Allen Roush

Florian Brand of Prime Intellect discusses how to evaluate agents when their tools and execution environments affect the result. The episode examines benchmark gaming, preventing shortcuts around scoring rules and the statistical limits of expensive evaluation runs. It also asks whether subjective impressions of reliability can be translated into useful measurements of model behavior. Hosted by Ravid Shwartz Ziv and Allen Roush. Watch the full conversation on the publisher’s YouTube channel.

Hosted by Ravid Shwartz Ziv, Allen Roush

Pleias co-founder Pierre-Carl Langlais describes language models built from documented open and public-domain sources alongside synthetic data. The conversation examines the SYNTH dataset, gaps in common web crawls, preservation of source material and ethical choices beyond copyright status. It also covers compact deployed models, benchmark incentives, reasoning-trace access and competing approaches to national AI development. Hosted by Ravid Shwartz Ziv and Allen Roush. Watch the full conversation on the publisher’s YouTube channel.

Hosted by Ravid Shwartz Ziv, Allen Roush

Dhruv Batra connects embodied AI research with browser agents at Yutori. He explains why navigation from pixels to actions links virtual robots and web interaction, then discusses reinforcement learning on live websites, simulation-to-reality gaps and teleoperation. The conversation also examines competing views of intelligence, claims about scaling and the implications of agent-mediated browsing for the web. Hosted by Ravid Shwartz Ziv and Allen Roush. Watch the full conversation on the publisher’s YouTube channel.

Hosted by Ravid Shwartz Ziv, Allen Roush