Topic: model weights

Damian Borth describes learning from trained neural networks themselves as a new source of training data. The conversation explores representations of model weights, transferring knowledge across architectures and tasks, and the possibility of specializing models with less computation by reusing what existing networks have learned.

Jacob Buckman examines long-context transformer architectures and the balance between model weights and recurrent state. Topics include windowed, grouped-query and latent attention, Power Retention, efficient GPU kernels and a coding model using recurrence. The conversation also discusses measuring how much useful information models extract from context.