Language model pre-training is an increasingly promising research approach in NLP [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12].
As pre-training uses unlabeled text, it can be combined with scaling model and dataset sizes to achieve better performance
or new capabilities [13]. For example, GPT-3 [12], a 175B parameter model trained on a large corpus of unlabeled text,
shows an impressive ability in few-shot learning thanks to scaling.
Dialog models [14, 15, 16], one of the most interesting applications of large language models, successfully take
advantage of Transformers’ ability to represent long-term dependencies in text [17, 18]. Similar to general language
models [13], Adiwardana et al. [17] show that dialog models are also well suited to model scaling. There is a strong
correlation between model size and dialog quality.
Inspired by these successes, we train LaMDA, a family of Transformer-based neural language models designed for
LaMDA- Language Models for Dialog Applications