Figure 1: Training loss over train tokens for the 7B,
13B, 33B, and 65 models. LLaMA-33B and LLaMA-
65B were trained on 1.4T tokens. The smaller models
were trained on 1.0T tokens. All models are trained
with a batch size of 4M tokens.

steps, and vary the learning rate and batch size with
the size of the model (see Table 2 for details).

2.4 Efﬁcient implementation
We make several optimizations to improve the train-
ing speed of our models. First, we use an efﬁcient
implementation of the causal multi-head attention
to reduce memory usage and runtime. This imple-
mentation, available in the xformers library,2 is
inspired by Rabe and Staats (2021) and uses the
backward from Dao et al. (2022). This is achieved
by not storing the attention weights and not com-
puting the key/query scores that are masked due to
the causal nature of the language modeling task.
LLaMA- Open and Efficient Foundation Language Models