7 Related work
Language models
are probability distributions
over sequences of words,
tokens or charac-
ters (Shannon, 1948, 1951). This task, often framed
as next token prediction, has long been considered a
core problem in natural language processing (Bahl
et al., 1983; Brown et al., 1990). Because Turing
(1950) proposed to measure machine intelligence
by using language through the “imitation game”,
language modeling has been proposed as a bench-
mark to measure progress toward artiﬁcial intelli-
gence (Mahoney, 1999).

Architecture. Traditionally,
language models
were based on n-gram count statistics (Bahl
et al., 1983), and various smoothing techniques
were proposed to improve the estimation of rare
events (Katz, 1987; Kneser and Ney, 1995). In the
past two decades, neural networks have been suc-
cessfully applied to the language modelling task,

4https://github.com/facebookresearch/metaseq/

tree/main/projects/OPT/chronicles

GPU Type

GPU Power
consumption

GPU-hours
LLaMA- Open and Efficient Foundation Language Models