5

Llama 1

See Touvron et al.
(2023)

Llama 2

A new mix of publicly
available online data

Training Data

Params

GQA

Tokens

LR

Context
Length

2k
2k
2k
2k
4k
4k
4k
4k

7B
13B
33B
65B
7B
13B
34B
70B

✗
✗
✗
✗

✗
✗
✓
✓

1.0T
1.0T
1.4T
1.4T
2.0T
2.0T
2.0T
2.0T

3.0 × 10−4
3.0 × 10−4
1.5 × 10−4
1.5 × 10−4
3.0 × 10−4
3.0 × 10−4
1.5 × 10−4
1.5 × 10−4

Table 1: Llama 2 family of models. Token counts refer to pretraining data only. All models are trained with
a global batch-size of 4M tokens. Bigger models — 34B and 70B — use Grouped-Query Attention (GQA) for
improved inference scalability.

Figure 5: Training Loss for Llama 2 models. We compare the training loss of the Llama 2 family of models.
We observe that after pretraining on 2T Tokens, the models still did not show any sign of saturation.
Llama2