CB dev acc 13 80.4 56 100.0 4 78.8 52 93% -2%
Anagrams 2 dev acc 2 40.2 10000 76.2 705 37.4 9295 93% -7%
Reversed Words dev acc 2 0.4 10000 1.5 660 0.3 9340 93% -26%
OpenBookQA test acc 8 65.4 500 58.1 31 65.9 469 94% 1%
ARC (Easy) test acc 11 70.1 2268 77.5 89 69.8 2179 96% 0%
Anagrams 1 dev acc 2 15.0 10000 49.8 327 13.8 9673 97% -8%
COPA dev acc 9 93.0 100 100.0 3 92.8 97 97% 0%
ARC (Challenge) test acc 12 51.6 1144 45.2 31 51.8 1113 97% 0%
HellaSwag dev acc 13 79.3 10042 86.2 152 79.2 9890 98% 0%
NQs test acc 11 29.9 3610 32.7 52 29.8 3558 99% 0%
Cycled Letters dev acc 2 38.6 10000 20.5 73 38.7 9927 99% 0%
SAT Analogies dev acc 9 65.8 374 100.0 2 65.6 372 99% 0%
Recent work has demonstrated substantial gains on many NLP tasks and
benchmarks by pre-training on a large corpus of text followed by fine-tuning on
a specific task. While typically task-agnostic in architecture, this method
still requires task-specific fine-tuning datasets of thousands or tens of
thousands of examples. By contrast, humans can generally perform a new language
task from only a few examples or from simple instructions - something which
current NLP systems still largely struggle to do. Here we show that scaling up
language models greatly improves task-agnostic, few-shot performance, sometimes
even reaching competitiveness with prior state-of-the-art fine-tuning
approaches. Specifically, we train GPT-3, an autoregressive language model with
175 billion parameters, 10x more than any previous non-sparse language model,
and test its performance in the few-shot setting. For all tasks, GPT-3 is
applied without any gradient updates or fine-tuning, with tasks and few-shot
demonstrations specified purely via text interaction with the model. GPT-3
achieves strong performance on many NLP datasets, including translation,
question-answering, and cloze tasks, as well as several tasks that require
on-the-fly reasoning or domain adaptation, such as unscrambling words, using a
novel word in a sentence, or performing 3-digit arithmetic. At the same time,
we also identify some datasets where GPT-3's few-shot learning still struggles,
as well as some datasets where GPT-3 faces methodological issues related to
training on large web corpora. Finally, we find that GPT-3 can generate samples
of news articles which human evaluators have difficulty distinguishing from
articles written by humans. We discuss broader societal impacts of this finding
and of GPT-3 in general.
Tom B. Brown Benjamin Mann Nick Ryder Melanie Subbiah Jared Kaplan Prafulla Dhariwal Arvind Neelakantan Pranav Shyam Girish Sastry Amanda Askell Sandhini Agarwal Ariel Herbert-Voss Gretchen Krueger Tom Henighan Rewon Child Aditya Ramesh Daniel M. Ziegler Jeffrey Wu Clemens Winter Christopher Hesse Mark Chen Eric Sigler Mateusz Litwin Scott Gray Benjamin Chess Jack Clark Christopher Berner Sam McCandlish Alec Radford Ilya Sutskever Dario Amodei