summaries: This paper describes a dataset consisting of ~14k multiple-choice questions drawn from many different fields across the humanities and science as well as professional disciplines such as law and medicine. It presents results for GPT-3 models (LMs trained on text corpora with document context) of different scales, as well as for the UnifiedQA model (seq2seq model trained on various QA datasets). Performance of these models is well below their performance on other benchmarks: not above chance for the smaller GPT-3 models, and under 50% average accuracy for the best models. This paper focuses on coming up with 57 different tasks and measure the performance of these large scale transformer models such as GPT3 on these different tasks. The main claims of this paper are to demonstrate these large-scale models still struggle to use the knowledge it has learned during the pretraining phase and these models struggle to on calculation-intensive tasks. Further one of the more important contributions of this work includes the massive multi-task dataset that comprises 57 different subjects. The paper proposes a benchmark for NLP models. The purpose of this test is to measure the model's knowledge in 57 topics covered by approx 15000 tasks in total, each formulated as a closed-form question in zero-shot and few-shot settings. Most of the tasks were taken from different human examination sets. Then, the paper provides results of experiments with the latest (GPT-3 and T5 based) models along with some quantitative and qualitative observations.
abstractText: We propose a new test to measure a text model’s multitask accuracy. The test covers 57 tasks including elementary mathematics, US history, computer science, law, and more. To attain high accuracy on this test, models must possess extensive world knowledge and problem solving ability. We find that while most recent models have near random-chance accuracy, the very largest GPT-3 model improves over random chance by almost 20 percentage points on average. However, on every one of the 57 tasks, the best models still need substantial improvements before they can reach expert-level accuracy. Models also have lopsided performance and frequently do not know when they are wrong. Worse, they still have nearrandom accuracy on some socially important subjects such as morality and law. By comprehensively evaluating the breadth and depth of a model’s academic and professional understanding, our test can be used to analyze models across many tasks and to identify important shortcomings.
authors.name: Dan Hendrycks
authors.name: Steven Basart
references.authors: M.G. Bellemare Y. Naddaf J. Veness M. Bowling
references.title: The arcade learning environment: An evaluation platform for general agents (extended abstract)
references.venue: J. Artif. Intell. Res.,
references.authors: Y. Bisk R. Zellers R.L. Bras J. Gao Y. Choi
references.title: Piqa: Reasoning about physical commonsense in natural language, 2019
references.authors: T.B. Brown B. Mann N. Ryder M. Subbiah J. Kaplan P. Dhariwal A. Neelakantan P. Shyam G. Sastry A. Askell S. Agarwal A. Herbert-Voss G. Krueger T. Henighan R. Child A. Ramesh D.M. Ziegler J. Wu C. Winter C. Hesse M. Chen E. Sigler M. Litwin S. Gray B. Chess J. Clark C. Berner S. McCandlish A. Radford I. Sutskever D. Amodei
references.title: Language models are few-shot learners, 2020
references.authors: P. Clark I. Cowhey O. Etzioni T. Khot A. Sabharwal C. Schoenick O. Tafjord
references.title: Think you have solved question answering? try arc, the ai2 reasoning
references.venue: challenge. ArXiv,
references.authors: J. Devlin M.-W. Chang K. Lee K. Toutanova. Bert
references.title: Pre-training of deep bidirectional transformers for language understanding
references.venue: ArXiv,
references.authors: R. Geirhos J.-H. Jacobsen C. Michaelis R. Zemel W. Brendel M. Bethge F.A. Wichmann
references.title: Shortcut learning in deep neural networks, 2020
references.authors: C. Guo G. Pleiss Y. Sun K.Q. Weinberger
references.title: On calibration of modern neural networks
references.authors: D. Hendrycks M. Mazeika T. Dietterich
references.title: Deep anomaly detection with outlier exposure
references.venue: ICLR,
references.authors: D. Hendrycks C. Burns S. Basart A. Critch J. Li D. Song J. Steinhardt
references.title: Aligning ai with shared human
references.authors: L. Huang R.L. Bras C. Bhagavatula Y. Choi
references.title: Cosmos qa: Machine reading comprehension with contextual commonsense reasoning, 2019
references.authors: J. Kaplan S. McCandlish T. Henighan T.B. Brown B. Chess R. Child S. Gray A. Radford J. Wu D. Amodei
references.title: Scaling laws for neural language models, 2020
references.authors: D. Khashabi T. Khot A. Sabharwal O. Tafjord P. Clark H. Hajishirzi
references.title: Unifiedqa: Crossing format boundaries with a single qa system, 2020
references.authors: T. Khot P. Clark M. Guerquin P. Jansen A. Sabharwal
references.title: Qasc: A dataset for question answering via sentence composition, 2019
references.authors: A. Kumar P. Liang T. Ma
references.title: Verified uncertainty calibration, 2019
references.authors: G. Lai Q. Xie H. Liu Y. Yang E. Hovy
references.title: Race: Large-scale reading comprehension dataset from examinations, 2017
references.authors: Z. Lan M. Chen S. Goodman K. Gimpel P. Sharma R. Soricut
references.title: Albert: A lite bert for self-supervised learning of language
references.venue: representations. ArXiv,
references.authors: Y. Liu M. Ott N. Goyal J. Du M. Joshi D. Chen O. Levy M. Lewis L. Zettlemoyer V. Stoyanov
references.title: Roberta: A robustly optimized bert pretraining approach
references.authors: T. Mihaylov P. Clark T. Khot A. Sabharwal
references.title: Can a suit of armor conduct electricity? a new dataset for open book question answering
references.authors: Y. Ovadia E. Fertig J. Ren Z. Nado D. Sculley S. Nowozin J.V. Dillon B. Lakshminarayanan J. Snoek
references.title: Can you trust your model’s uncertainty? Evaluating predictive uncertainty under dataset shift
references.venue: NeurIPS,
references.authors: F. Petroni T. Rocktäschel P. Lewis A. Bakhtin Y. Wu A.H. Miller S. Riedel
references.title: Language models as knowledge bases?, 2019
references.authors: A. Radford J. Wu R. Child D. Luan D. Amodei I. Sutskever
references.title: Language models are unsupervised multitask learners
references.authors: C. Raffel N. Shazeer A. Roberts K. Lee S. Narang M. Matena Y. Zhou W. Li P.J. Liu
references.title: Exploring the limits of transfer learning with a unified text-to-text transformer, 2019
references.authors: M. Richardson C.J. Burges E. Renshaw
references.title: MCTest: A challenge dataset for the open-domain machine comprehension of text
references.venue: In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing,
references.authors: A.B. Sai A.K. Mohankumar M.M. Khapra
references.title: A survey of evaluation metrics used for nlg systems
references.authors: A. Turing
references.title: Computing machinery and intelligence
references.authors: A. Wang A. Singh J. Michael F. Hill O. Levy S.R. Bowman
references.title: Glue: A multi-task benchmark and analysis platform for natural language understanding, 2018
references.authors: A. Wang Y. Pruksachatkun N. Nangia A. Singh J. Michael F. Hill O. Levy S.R. Bowman
references.title: Superglue: A stickier benchmark for general-purpose language understanding systems, 2019
references.authors: R. Zellers A. Holtzman Y. Bisk A. Farhadi Y. Choi
references.title: Hellaswag: Can a machine really finish your sentence
references.authors: R. Zellers A. Holtzman E. Clark L. Qin A. Farhadi Y. Choi
references.title: Evaluating machines by their real-world language use, 2020
references.authors: See Brown
references.title: 2020) for a previous discussion of contamination showing that the phenomena hardly affects performance. To reduce the probability that future models encounter exact questions during test-time, we will provide a list of question sources
sections.heading: 1 INTRODUCTION
sections.text: Natural Language Processing (NLP) models have achieved superhuman performance on a number of recently proposed benchmarks. However, these models are still well below human level performance for language understanding as a whole, suggesting a disconnect between our benchmarks and the actual capabilities of these models. The General Language Understanding Evaluation benchmark (GLUE) (Wang et al., 2018) was introduced in 2018 to evaluate performance on a wide range of NLP tasks, and top models achieved superhuman performance within a year. To address the shortcomings of GLUE, researchers designed the SuperGLUE benchmark with more difficult tasks (Wang et al., 2019). About a year since the release of SuperGLUE, performance is again essentially human-level (Raffel et al., 2019). While these benchmarks evaluate linguistic skills more than overall language understanding, an array of commonsense benchmarks have been proposed to measure basic reasoning and everyday knowledge (Zellers et al., 2019; Huang et al., 2019; Bisk et al., 2019). However, these recent benchmarks have similarly seen rapid progress (Khashabi et al., 2020). Overall, the near human-level performance on these benchmarks suggests that they are not capturing important facets of language understanding.
  Transformer models have driven this recent progress by pretraining on massive text corpora, including all of Wikipedia, thousands of books, and numerous websites. These models consequently see extensive information about specialized topics, most of which is not assessed by existing NLP benchmarks. It consequently remains an open question just how capable current language models are at learning and applying knowledge from many domains.
  To bridge the gap between the wide-ranging knowledge that models see during pretraining and the existing measures of success, we introduce a new benchmark for assessing models across a diverse set of subjects that humans learn. We design the benchmark to measure knowledge acquired during pretraining by evaluating models exclusively in zero-shot and few-shot settings. This makes the benchmark more challenging and more similar to how we evaluate humans. The benchmark covers 57 subjects across STEM, the humanities, the social sciences, and more. It ranges in difficulty from an elementary level to an advanced professional level, and it tests both world knowledge and problem solving ability. Subjects range from traditional areas, such as mathematics and history, to more
  Few Shot Prompt and Predicted Answer
sections.heading: How many numbers are in the list 25, 26, ..., 100?
sections.text: (A) 75 (B) 76 (C) 22 (D) 23
sections.heading: Answer: B
sections.text: Compute i + i2 + i3 + ··· + i 258 + i259. (A) -1 (B) 1 (C) i (D) -i
sections.heading: Answer: A
sections.text: If 4 daps = 7 yaps, and 5 yaps = 3 baps, how many daps equal 42 baps? (A) 28 (B) 21 (C) 40 (D) 30 Answer: C␣
  The following are multiple choice questions about high school mathematics.
  (a) An example of few-shot learning and inference using GPT-3. The blue underlined bold text is the autocompleted response from GPT-3, while the preceding text is the user-inputted prompt. In this 2-shot learning example, there are two instruction examples and one initially incomplete example. On average, GPT-3 has low accuracy on high school mathematics questions.
  Small Medium Large X-Large Model Size
  20
  30
  40
  50
  60
  70
  80
  90
  Pe rfo
  rm an
  ce (%
  )
  GPT-3 Few Shot Test Performance Commonsense Linguistics Knowledge (Ours)
  (b) Performance on a commonsense benchmark (HellaSwag), a linguistic understanding benchmark (SuperGLUE), and the massive multitask test. On previous benchmarks, smaller models start well above random chance levels and exhibit more continuous improvements with model size increases, but on our test, GPT-3 moves beyond random chance with the largest model.
  specialized areas like law and ethics (Hendrycks et al., 2020). The granularity and breadth of the subjects makes the benchmark ideal for identifying a model’s blind spots.
  We find that meaningful progress on our benchmark has only become possible in recent months. In particular, few-shot models up to 13 billion parameters (Brown et al., 2020) achieve random chance performance of 25% accuracy, but the 175 billion parameter GPT-3 model reaches a much higher 43.9% accuracy (see Figure 1b). On the other hand, unlike human professionals GPT-3 does not excel at any single subject. Instead, we find that performance is lopsided, with GPT-3 having almost 70% accuracy for its best subject but near-random performance for several other subjects.
  Our results indicate that while recent advances have been impressive, state-of-the-art models still struggle at learning and applying knowledge from pretraining. The tasks with near-random accuracy include calculation-heavy subjects such as physics and mathematics and subjects related to human values such as law and morality. This second weakness is particularly concerning because it will be important for future models to have a strong understanding of what is legal and what is ethical. Worryingly, we also find that GPT-3 does not have an accurate sense of what it does or does not know since its average confidence can be up to 24% off from its actual accuracy. We comprehensively evaluate the breadth and depth of a model’s text understanding by covering numerous topics that humans are incentivized to learn. Since our test consists in 57 tasks, it can be used to analyze aggregate properties of models across tasks and to track important shortcomings. The test and code is available at github.com/hendrycks/test.
sections.heading: 2 RELATED WORK
sections.text: Pretraining. The dominant paradigm in NLP is to pretrain large models on massive text corpora including educational books and websites. In the process, these models are exposed to information about a wide range of topics. Petroni et al. (2019) found that recent models learn enough information from pretraining that they can serve as knowledge bases. However, no prior work has comprehensively measured the knowledge models have across many real-world domains.
  Until recently, researchers primarily used fine-tuned models on downstream tasks (Devlin et al., 2019). However, larger pretrained models like GPT-3 (Brown et al., 2020) have made it possible to achieve competitive performance without fine-tuning by using few-shot learning, which removes the need for a large fine-tuning set. With the advent of strong zero-shot and few-shot learning, it is now possible to curate a diverse set of tasks for evaluation and remove the possibility of models on “spurious cues” (Geirhos et al., 2020; Hendrycks et al., 2019b) in a dataset to achieve high performance.
  Benchmarks. Many recent benchmarks aim to assess a model’s general world knowledge and basic reasoning ability by testing its “commonsense.” A number of commonsense benchmarks have been
  proposed in the past year, but recent models are already nearing human-level performance on several of these, including HellaSwag (Zellers et al., 2019), Physical IQA (Bisk et al., 2019), and CosmosQA (Huang et al., 2019). By design, these datasets assess abilities that almost every child has. In contrast, we include harder specialized subjects that people must study to learn.
  Some researchers have suggested that the future of NLP evaluation should focus on Natural Language Generation (NLG) (Zellers et al., 2020), an idea that reaches back to the Turing Test (Turing, 1950). However, NLG is notoriously difficult to evaluate and lacks a standard metric (Sai et al., 2020). Consequently, we instead create a simple-to-evaluate test that measures classification accuracy on multiple choice questions.
  While several question answering benchmarks exist, they are comparatively limited in scope. Most either cover easy topics like grade school subjects for which models can already achieve strong performance (Clark et al., 2018; Khot et al., 2019; Mihaylov et al., 2018; Clark et al., 2019), or are focused on linguistic understanding in the form of reading comprehension (Lai et al., 2017; Richardson et al., 2013). In contrast, we include a wide range of difficult subjects that go far beyond linguistic understanding.
sections.heading: 3 A MULTITASK TEST
sections.text: We create a massive multitask test consisting of multiple-choice questions from various branches of knowledge. The test spans subjects in the humanities, social sciences, hard sciences, and other areas that are important for some people to learn. There are 57 tasks in total, which is also the number of Atari games (Bellemare et al., 2013), all of which are listed in Appendix B. The questions in the dataset were manually collected by graduate and undergraduate students from freely available sources online. These include practice questions for tests such as the Graduate Record Examination and the United States Medical Licensing Examination. It also includes questions designed for undergraduate courses and questions designed for readers of Oxford University Press books. Some tasks cover a subject, like psychology, but at a specific level of difficulty, such as “Elementary,” “High School,” “College,” or “Professional.” For example, the “Professional Psychology” task draws on questions from freely available practice questions for the Examination for Professional Practice in Psychology, while the “High School Psychology” task has questions like those from Advanced Placement Psychology examinations.
  We collected 15908 questions in total, which we split into a few-shot development set, a validation set, and a test set. The few-shot development set has 5 questions per subject, the validation set may be used for selecting hyperparameters and is made of 1540 questions, and the test set has 14079 questions. Each subject contains 100 test examples at the minimum, which is longer than most exams designed to assess people.
  Human-level accuracy on this test varies. Unspecialized humans from Amazon Mechanical Turk obtain 34.5% accuracy on this test. Meanwhile, expert-level performance can be far higher. For example, real-world test-taker human accuracy at the 95th percentile is around 87% for US Medical Licensing Examinations, and these questions make up our “Professional Medicine” task. If we take the 95th percentile human test-taker accuracy for exams that build up our test, and if we make an educated guess when such information is unavailable, we then estimate that expert-level accuracy is approximately 89.8%.
  Since our test aggregates different subjects and several levels of difficulty, we measure more than straightforward commonsense or narrow linguistic understanding. Instead, we measure arbitrary
  When you drop a ball from rest it accelerates downward at 9.8 m/s². If you instead throw it downward assuming no air resistance its acceleration immediately after leaving your hand is (A) 9.8 m/s² (B) more than 9.8 m/s² (C) less than 9.8 m/s² (D) Cannot say unless the speed of throw is given. C on ce pt ua l Ph ys ic s
  C ol
  le ge
  M at
  he m
  at ic s In the complex z-plane, the set of points satisfying the equation z² = |z|² is a (A) pair of points (B) circle (C) half-line (D) line
  Figure 4: Examples from the Conceptual Physics and College Mathematics STEM tasks.
  real-world text understanding. Since models are pretrained on the Internet, this enables us to test how well they can extract useful knowledge from massive corpora. Future models that use this test could be single models or a mixture of experts model. To succeed at our test, future models should be well-rounded, possess extensive world knowledge, and develop expert-level problem solving ability. These properties make the test likely to be an enduring and informative goalpost.
sections.heading: 3.1 HUMANITIES
sections.text: The humanities is a group of disciplines that make use of qualitative analysis and analytic methods rather than scientific empirical methods. Branches of the humanities include law, philosophy, history, and so on (Appendix B). Mastering these subjects requires a variety of skills. For example, legal understanding requires knowledge of how to apply rules and standards to complex scenarios, and also provide answers with stipulations and explanations. We illustrate this in Figure 2. Legal understanding is also necessary for understanding and following rules and regulations, a necessary capability to constrain open-world machine learning models. For philosophy, our questions cover concepts like logical fallacies, formal logic, and famous philosophical arguments. It also covers moral scenarios, including questions from the ETHICS dataset (Hendrycks et al., 2020) that test a model’s understanding of normative statements through predicting widespread moral intuitions about diverse everyday scenarios. Finally, our history questions cover a wide range of time periods and geographical locations, including prehistory and other advanced subjects.
sections.heading: 3.2 SOCIAL SCIENCE
sections.text: Social science includes branches of knowledge that examine human behavior and society. Subject areas include economics, sociology, politics, geography, psychology, and so on. See Figure 3 for an example question. Our economics questions include microeconomics, macroeconomics, and econometrics, and cover different types of problems, including questions that require a mixture of world knowledge, qualitative reasoning, or quantitative reasoning. We also include important but more esoteric topics such as security studies in order to test the boundaries of what is experienced and learned during pretraining. Social science also includes psychology, a field that may be especially important for attaining a nuanced understanding of humans.
sections.heading: 3.3 SCIENCE, TECHNOLOGY, ENGINEERING, AND MATHEMATICS (STEM)
sections.text: STEM subjects include physics, computer science, mathematics, and more. Two examples are shown in Figure 4. Conceptual physics tests understanding of simple physics principles and may be thought
  of as a harder version of the physical commonsense benchmark Physical IQA (Bisk et al., 2019). We also test mathematical problem solving ability at various levels of difficulty, from the elementary to the college level. College mathematics questions, like those found on the GRE mathematics subject test, often require chains of reasoning and abstract knowledge. To encode mathematics expressions, we use LaTeX or symbols such as * and ˆ for multiplication and exponentiation respectively. STEM subjects require knowledge of empirical methods, fluid intelligence, and procedural knowledge.
sections.heading: 3.4 OTHER
sections.text: There is a long tail of subjects that either do not neatly fit into any of the three preceding categories or for which there are not thousands of freely available questions. We put these subjects into Other. This section includes the Professional Medicine task, which has difficult questions that require humans many years of study to master. An example is depicted in Figure 5. This section also contains business topics like finance, accounting, and marketing, as well as knowledge of global facts. The latter includes statistics about poverty in different countries over time, which may be necessary for having an accurate model of the world internationally.
sections.heading: 4 EXPERIMENTS
sections.heading: 4.1 SETUP
sections.text: Assessment and Models. To measure performance on our multitask test, we compute the classification accuracy across all examples and tasks. We evaluate GPT-3 (Brown et al., 2020) and UnifiedQA (Khashabi et al., 2020). For GPT-3 we use the OpenAI API, which provides access to four model variants, “Ada,” “Babbage,” “Curie,” and “Davinci,” which we refer to as “Small” (2.7 billion parameters), “Medium” (6.7 billion), “Large” (13 billion) and “X-Large” (175 billion). UnifiedQA uses the T5 (Raffel et al., 2019) text-to-text backbone and is fine-tuned on previously proposed question answering datasets (Lai et al., 2017), where the prediction is the class with the highest token overlap with UnifiedQA’s text output. Since UnifiedQA is fine-tuned on other datasets, we evaluate it without any further tuning to assess its transfer accuracy. We also fine-tune RoBERTa-base, ALBERT-xxlarge, and GPT-2 on UnifiedQA training data and our dev+val set. We primarily focus on UnifiedQA and GPT-3 in the rest of this document, but additional discussion of RoBERTa, ALBERT, and GPT-2 is in Appendix A.
  Few-Shot Prompt. We feed GPT-3 prompts like that shown in Figure 1a. We begin each prompt with “The following are multiple choice questions (with answers) about [subject].” For zero-shot evaluation, we append the question to the prompt. For few-shot evaluation, we add up to 5 demonstration examples with answers to the prompt before appending the question. All prompts end with “Answer: ”. The model then produces probabilities for the tokens “A,” “B,” “C,” and “D,” and we treat the highest probability option as the prediction. For consistent evaluation, we create a dev set with 5 fixed few-shot examples for each subject.
sections.heading: 4.2 RESULTS
sections.text: Model Size and Accuracy. We compare the few-shot accuracy of each GPT-3 size in Table 1. We find that the three smaller GPT-3 models have near random accuracy (around 25%). In contrast, we find that the X-Large 175 billion parameter GPT-3 model performs substantially better than random, with an accuracy of 43.9%. We also find qualitatively similar results in the zero-shot setting. While the smaller models have around 25% zero-shot accuracy, Figure 10 in Appendix A shows that the largest GPT-3 model has a much higher zero-shot accuracy of about 37.7%. Brown et al. (2020) also observe that larger GPT-3 models perform better, though progress tends to be steadier. In Figure 1b we show that non-random accuracy on the multitask test emerged with recent large few-shot models compared to datasets that assess commonsense and linguistic understanding.
  To test the usefulness of fine-tuning instead of few-shot learning, we also evaluate UnifiedQA models. UnifiedQA has the advantage of being fine-tuned on other question answering datasets, unlike GPT-3. We assess UnifiedQA by evaluating its transfer performance without any additional fine-tuning. The largest UnifiedQA model we test has 11 billion parameters, which is slightly smaller than GPT-3 Large. Nevertheless, we show in Table 1 that it attains 48.9%
  accuracy. This performs better than the few-shot GPT-3 X-Large model, despite UnifiedQA have an order of magnitude fewer parameters. We also find that even the smallest UnifiedQA variant, with just 60 million parameters, has approximately 29.3% accuracy. These results suggest that while model size is a key component for achieving strong performance, fine-tuning also helps.
  Comparing Disciplines. Using our test, we discover that GPT-3 and UnifiedQA have lopsided performance and several substantial knowledge gaps. Figure 6 shows the accuracy of GPT-3 (fewshot) and UnifiedQA for all 57 tasks. It shows the both models are below expert-level performance for all tasks, with GPT-3’s accuracy ranging from 69% for US Foreign Policy to 26% for College Chemistry. UnifiedQA does best on marketing, with an accuracy of 82.5%.
  Overall, models do poorly on highly procedural problems. Figure 6 shows that calculation-heavy STEM subjects tend to have low accuracy compared to verbal subjects. For GPT-3, 9 out of the 10
  Figure 7: GPT-3’s completion for two prompts testing knowledge of the order of operations. The blue underlined bold text is the autocompleted response from GPT-3. While it knows about the order of operations, it sometimes does not know how to apply its knowledge.
  lowest-accuracy tasks are STEM subjects that emphasize mathematics or calculations. We speculate that is in part because GPT-3 acquires declarative knowledge more readily than procedural knowledge. For example, many questions in Elementary Mathematics require applying the order of operations for arithmetic, which is described by the acronym PEMDAS (Parentheses Exponents Multiplication Division Addition Subtraction). In Figure 7, we confirm that GPT-3 is aware of the acronym PEMDAS. However, it does not consistently apply PEMDAS to actual problems. On the other hand, procedural understanding is not its only weak point. We find that some verbal tasks such as Moral Scenarios from Hendrycks et al. (2020) and Professional Law also have especially low accuracy.
  Our test also shows that GPT-3 acquires knowledge quite unlike humans. For example, GPT-3 learns about topics in a pedagogically unusual order. GPT-3 does better on College Medicine (47.4%) and College Mathematics (35.0%) than calculation-heavy Elementary Mathematics (29.9%). GPT-3 demonstrates unusual breadth, but it does not master a single subject. Meanhwhile we suspect humans have mastery in several subjects but not as much breadth. In this way, our test shows that GPT-3 has many knowledge blindspots and has capabilities that are lopsided.
  Calibration. We should not trust a model’s prediction unless the model is calibrated, meaning that its confidence is a good estimate of the actual probability the prediction is correct. However, large neural networks are often miscalibrated (Guo et al., 2017), especially under distribution shift (Ovadia et al., 2019). We evaluate the calibration of GPT-3 by testing how well its average confidence estimates its actual accuracy for each subject. We show the results in Figure 8, which demonstrates that GPT-3 is uncalibrated. In fact, its confidence is only weakly related to its actual accuracy in the zero-shot setting, with the difference between its accuracy and confidence reaching up to 24% for some subjects. Another calibration measure is the Root Mean Squared (RMS) calibration error (Hendrycks et al., 2019a; Kumar et al., 2019). Many tasks have miscalibrated predictions, such as Elementary Mathematics which has a zero-shot RMS calibration error of 19.4%. Models are only somewhat more calibrated in the few-shot setting, as shown in Appendix A. These results suggest that model calibration has wide room for improvement.
sections.heading: 5 DISCUSSION
sections.text: Multimodal Understanding. While text is capable of conveying an enormous number of concepts about the world, many important concepts are conveyed mainly through other modalities, such as images, audio, and physical interaction (Bisk et al., 2020). Existing large-scale NLP models, such as GPT-3, do not incorporate multimodal information, so we design our benchmark to capture a diverse array of tasks in a text-only format. However, as models gain the ability to process multimodal inputs, benchmarks should be designed to reflect this change. One such benchmark could be a “Turk Test,” consisting of Amazon Mechanical Turk Human Intelligence Tasks. These are well-defined tasks that require models to interact with flexible formats and demonstrate multimodal understanding.
  The Internet as a Training Set. A major distinction between our benchmark and previous multitask NLP benchmarks is that we do not require large training sets. Instead, we assume that models have acquired the requisite knowledge from reading vast quantities of diverse text from the Internet. This
  process is typically called pretraining, but it can be thought of as training in its own right, where the downstream evaluation is demonstrating whatever knowledge we would expect a human to pick up from reading the same text.
  This motivates us to propose a methodological change so that models are trained more like how humans learn. While most previous machine learning benchmarks have models learn from a large question bank, humans primarily learn new subjects by reading books and listening to others talk about the topic. For specialized subjects such as Professional Law, massive legal corpora are available, such as the 164-volume legal encyclopedia Corpus Juris Secundum, but there are fewer than 5,000 multistate bar exam questions available. Learning the entire law exclusively through a small number of practice tests is implausible, so future models must learn more during pretraining.
  For this reason we assess pretrained models in a zero-shot, few-shot, or transfer setting and we provide a dev, val, and test set for each task. The dev set is used for few-shot prompts, the val set could be used for hyperparameter tuning, and the test set is used to compute the final accuracy. Importantly, the format of our evaluation is not identical to the format in which information is acquired during pretraining. This has the benefit of obviating concerns about spurious training set annotation artifacts (Geirhos et al., 2020; Hendrycks et al., 2019b) and is in stark contrast to the previous paradigm of identically distributed training and test sets. This change also enables collecting a much more extensive and diverse set of tasks for evaluation. We anticipate our methodology becoming more widespread as models improve at extracting information from diverse online sources.
  Model Limitations. We find that current large-scale Transformers have wide room for improvement. They are notably poor at modeling human (dis)approval, as evident by the low performance on the Professional Law and Moral Scenarios tasks. For future systems to be aligned with human values, high performance on these tasks is crucial (Hendrycks et al., 2020), so future research should especially aim to increase accuracy on these tasks. Models also have difficulty performing calculations, so much so that they exhibit poor performance on Elementary Mathematics and many other STEM subjects with “plug and chug” problems. Additionally, they do not match expert-level performance (90%) on any subject, so for all subjects it is subhuman. On average, models are only now starting to move beyond random-chance accuracy levels.
  Addressing these shortcomings may be challenging. To illustrate this, we attempted to create a better Professional Law model by pretraining on specialized data but achieved only limited success. We collected approximately 2,000 additional Professional Law training examples. After fine-tuning a RoBERTa-base model (Liu et al., 2019) using this custom training set, our model attained 32.8% test accuracy. To test the impact of additional specialized training data, we also had RoBERTa continue pretraining on approximately 1.6 million legal case summaries using Harvard’s Law Library case law corpus case.law, but after fine-tuning it only attained 36.1% accuracy. This suggests that while additional pretraining on relevant high quality text can help, it may not be enough to substantially increase the performance of current models.
  It is unclear whether simply scaling up existing language models will solve the test. Current understanding indicates that a 10× increase in model size must be accompanied by an approximate 5× increase in data (Kaplan et al., 2020). Aside from the tremendous expense in creating multi-trillion parameter language models, data may also become a bottleneck, as there is far less written about esoteric branches of knowledge than about everyday situations.
sections.heading: 6 CONCLUSION
sections.text: We introduced a new test that measures how well text models can learn and apply knowledge encountered during pretraining. By covering 57 subjects at varying levels of difficulty, the test assesses language understanding in greater breadth and depth than previous benchmarks. We found that it has recently become possible for models to make meaningful progress on the test, but that state-of-the-art models have lopsided performance and rarely excel at any individual task. We also showed that current models are uncalibrated and have difficulty with tasks that require calculations. Worryingly, models also perform especially poorly on socially relevant subjects including morality and law. Our expansive test can help researchers pinpoint important shortcomings of models, making it easier to gain a clearer picture of state-of-the-art capabilities.
sections.heading: ACKNOWLEDGEMENTS
sections.text: We would like to thank the following for their helpful comments: Oyvind Tafjord, Jan Leike, David Krueger, Alex Tamkin, Girish Sastry, and Henry Zhu. DH is supported by the NSF GRFP Fellowship and an Open Philanthropy Project Fellowship. This research was also supported by the NSF Frontier Award 1804794.
sections.heading: A ADDITIONAL ANALYSIS
sections.text: This appendix includes figures with sorted results (Figure 9), few-shot examples vs. accuracy (Figure 10), and few-shot calibration (Figure 11). It also includes sections on fine-tuning, error analysis, and format sensitivity.
  0 20 40 60 80 100
  Accuracy (%)
  College Chemistry Moral Scenarios College Physics
  High School Physics High School Mathematics
  Formal Logic Elementary Mathematics
  Abstract Algebra High School Statistics
  Machine Learning Econometrics
  High School Chemistry Professional Accounting
  Professional Law College Mathematics
  Professional Medicine Conceptual Physics
  Global Facts High School Comp Sci
  Medical Genetics High School Macroeconomics High School Microeconomics
  Moral Disputes Professional Psychology
  College Biology Virology
  College Comp Sci Business Ethics
  Nutrition College Medicine
  Anatomy Clinical Knowledge
  Logical Fallacies High School Biology
  Public Relations Astronomy
  Electrical Engineering Human Aging
  Philosophy Security Studies
  Prehistory High School US History
  Sociology High School European History
  Human Sexuality Jurisprudence
  World Religions International Law
  High School World History Management
  Computer Security High School Geography
  High School Gov't and Politics Marketing
  Miscellaneous High School Psychology
  US Foreign Policy GPT-3 Results
  Random Chance
  0 10 20 30 40 50 60 70 80 90 100
  Accuracy (%)
  Moral Scenarios Formal Logic
  Abstract Algebra Econometrics
  High School Mathematics College Physics
  Machine Learning High School Statistics
  College Chemistry Elementary Mathematics
  College Mathematics High School Chemistry
  Global Facts Professional Law Medical Genetics
  Professional Accounting College Biology
  High School Physics Anatomy
  College Comp Sci Conceptual Physics
  College Medicine Virology
  Professional Medicine Astronomy
  High School Macroeconomics Electrical Engineering
  Professional Psychology Security Studies
  Human Sexuality Nutrition
  High School Comp Sci Prehistory
  Clinical Knowledge High School Biology
  Human Aging High School Microeconomics
  Philosophy Public Relations World Religions Moral Disputes
  Logical Fallacies High School European History
  High School US History High School World History
  Miscellaneous US Foreign Policy
  Computer Security Sociology
  International Law High School Geography
  Jurisprudence Business Ethics
  High School Psychology Management
  High School Gov't and Politics Marketing
  UnifiedQA Results
  Random Chance
  Figure 9: On the left are GPT-3 few shot accuracies for all of the 57 tasks. On the right are UnifiedQA transfer accuracies for all of the 57 tasks. For both models, capabilities are lopsided.
sections.heading: A.1 ANALYSIS WITH MORE FINE-TUNED MODELS
sections.text: We primarily analyzed models with more than 10 billion parameters in the main body of the paper. For this section, we analyze smaller models including RoBERTa-base (125 million parameters) (Liu
  et al., 2019), ALBERT-xxlarge (223 million parameters) (Lan et al., 2020), and GPT-2 (1,558 million parameters) (Radford et al., 2019). Models are fine-tuned to predict one of four classes using the UnifiedQA MCQ questions and using our dev+val set. We test on our multitask test set.
  We observe that these smaller models can attain better-than-random accuracy. RoBERTa-base attains an overall accuracy of 27.9%, with 27.9% accuracy for the humanities, 28.8% for social sciences, 27.0% for STEM, and 27.7% for other. ALBERT-xxlarge attains an accuracy of 27.1%, with 27.2% accuracy for the humanities, 25.7% for the social sciences, 27.7% for STEM, and 27.9% for other. GPT-2 attains an accuracy of 32.4%, with 32.8% accuracy for the humanities, 33.3% for the social sciences, 30.2% for STEM, and 33.1% for other.
  Compare this to UnifiedQA’s smallest variant, which has just 60 million parameters and approximately 29.3% accuracy. It obtains higher accuracy than RoBERTa and ALBERT, even though it has fewer parameters. This suggests that its larger pretraining dataset enables higher accuracy. Likewise, UnifiedQA with 3 billion parameters attains 43.7%, while the similarly sized GPT-2 model with 1.5 billion parameters attains 32.4% accuracy. This again suggests that T5’s larger pretraining dataset size (and therefore UnifiedQA’s pretraining dataset size) can increase accuracy.
sections.heading: A.2 ERROR ANALYSIS
sections.text: We qualitatively analyze when GPT-3 makes high confidence mistakes. We find that while many of these mistakes were clearly wrong, many were mistakes that a human might make. For example, one question it got wrong was “How many chromosomes do all human somatic cells contain?” The correct answer is 46, while few-shot GPT-3 predicted 23 with confidence 97.5%. This answer would have been correct if the question asked about the number of pairs of chromosomes. Similarly, many of its other high confidence mistakes were also correct answers to slightly different questions.
sections.heading: A.3 FORMAT SENSITIVITY
sections.text: While different question formatting choices often lead to similar GPT-3 accuracies, we find that UnifiedQA is more sensitive. UnifiedQA’s input format is of the form
  QUESTION1 \\n (A) CHOICE1 (B) CHOICE2 (C) CHOICE3 (D) CHOICE4</s>
  where questions and choices are normalized and made lowercase. If we remove the </s> from the input, accuracy declines by several percentage points.
sections.heading: B TEST DETAILS
sections.heading: B.1 TASK DESCRIPTIONS AND EXAMPLES
sections.text: We provide analysis of question length and difficulty in Figure 12. We list all tasks and the topics they test in Table 2. We also provide an example for each task starting with Figure 14.
sections.heading: B.2 EXACT QUESTION AND ANSWER CONTAMINATION
sections.text: Since language models train on vast text corpora, there is some chance that they have seen the exact question and answer during pretraining. If they memorized the exact question and answer, then they would attain higher accuracy than their true ability. Likewise, a question’s entropy would be especially low if it were memorized. Memorized questions and answers should have low entropy and
  high accuracy. However, in Figure 13, we see that accuracy and question entropy are not positively correlated, suggesting that the test’s low-entropy questions do not correspond to memorized (and thereby correctly predicted) answers. This suggests that our exact questions were not memorized. However, during pretraining models encountered text related to our questions through processing Wikipedia. We also note that most of our questions came from PDFs or websites where questions and answers are on separate pages.
  See Brown et al. (2020) for a previous discussion of contamination showing that the phenomena hardly affects performance. To reduce the probability that future models encounter exact questions during test-time, we will provide a list of question sources.
title: MEASURING MASSIVE MULTITASK LANGUAGE UNDERSTANDING