LLaMA: Open and Efﬁcient Foundation Language Models

Hugo Touvron∗, Thibaut Lavril∗, Gautier Izacard∗, Xavier Martinet

Marie-Anne Lachaux, Timothee Lacroix, Baptiste Rozière, Naman Goyal

Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin

Edouard Grave∗, Guillaume Lample∗

Meta AI

Abstract

We introduce LLaMA, a collection of founda-
tion language models ranging from 7B to 65B
parameters. We train our models on trillions
of tokens, and show that it is possible to train
state-of-the-art models using publicly avail-
able datasets exclusively, without resorting
to proprietary and inaccessible datasets.
In
particular, LLaMA-13B outperforms GPT-3
(175B) on most benchmarks, and LLaMA-
65B is competitive with the best models,
Chinchilla-70B and PaLM-540B. We release
all our models to the research community1.

Introduction
LLaMA- Open and Efficient Foundation Language Models