Recurrence and Attention for Long-Context Transformers

Jacob Buckman examines long-context transformer architectures and the balance between model weights and recurrent state. Topics include windowed, grouped-query and latent attention, Power Retention, efficient GPU kernels and a coding model using recurrence. The conversation also discusses measuring how much useful information models extract from context.