Topic: Transformer

Modern artificial intelligence (AI) systems are powered by foundation models. This paper presents a new set of foundation models, called Llama 3. It is a herd of language models that natively support multilinguality, coding, reasoning, and tool usage. Our largest model is a dense Transformer with 405B parameters and a context window of up to 128K tokens. This paper presents an extensive empirical evaluation of Llama 3. We find that Llama 3 delivers comparable quality to leading language models such as GPT-4 on a plethora of tasks. We publicly release Llama 3, including pre-trained and post-tra

Mounting evidence suggests that traveling waves are a key dynamical motif in biological neural information processing systems [Muller et al., 2018], yet their precise computational role remains under investigation. While the Transformer architecture [Vaswani et al., 2023] has achieved state-of-the-art performance across various data modalities, it relies heavily on an explicit `context’ – processing the entire input sequence simultaneously. This approach is both biologically implausible and computationally inefficient, resulting in quadratic complexity relative to sequence length. Recent inter