Unlike Chinchilla, PaLM, or GPT-3, we only
use publicly available data, making our work com-
patible with open-sourcing, while most existing
models rely on data which is either not publicly
available or undocumented (e.g. “Books – 2TB” or
“Social media conversations”). There exist some
exceptions, notably OPT (Zhang et al., 2022),
GPT-NeoX (Black et al., 2022), BLOOM (Scao
et al., 2022) and GLM (Zeng et al., 2022), but none
that are competitive with PaLM-62B or Chinchilla.

In the rest of this paper, we present an overview
of the modiﬁcations we made to the transformer
architecture (Vaswani et al., 2017), as well as our
training method. We then report the performance of
our models and compare with others LLMs on a set
of standard benchmarks. Finally, we expose some
of the biases and toxicity encoded in our models,
using some of the most recent benchmarks from
the responsible AI community.

3
2
0
2

 

b
e
F
7
2

 

 
 
]
L
C
.
s
c
[
 
 

1
v
1
7
9
3
1

.

2
0
3
2
:
v
i
X
r
a
LLaMA- Open and Efficient Foundation Language Models