summaries: This paper describes a dataset consisting of ~14k multiple-choice questions drawn from many different fields across the humanities and science as well as professional disciplines such as law and medicine. It presents results for GPT-3 models (LMs trained on text corpora with document context) of different scales, as well as for the UnifiedQA model (seq2seq model trained on various QA datasets). Performance of these models is well below their performance on other benchmarks: not above chance for the smaller GPT-3 models, and under 50% average accuracy for the best models.

summaries: This paper focuses on coming up with 57 different tasks and measure the performance of these large scale transformer models such as GPT3 on these different tasks. The main claims of this paper are to demonstrate these large-scale models still struggle to use the knowledge it has learned during the pretraining phase and these models struggle to on calculation-intensive tasks. Further one of the more important contributions of this work includes the massive multi-task dataset that comprises 57 different subjects.

summaries: The paper proposes a benchmark for NLP models. The purpose of this test is to measure the model's knowledge in 57 topics covered by approx 15000 tasks in total, each formulated as a closed-form question in zero-shot and few-shot settings. Most of the tasks were taken from different human examination sets. Then, the paper provides results of experiments with the latest (GPT-3 and T5 based) models along with some quantitative and qualitative observations.

abstractText: We propose a new test to measure a text model’s multitask accuracy. The test covers 57 tasks including elementary mathematics, US history, computer science, law, and more. To attain high accuracy on this test, models must possess extensive world knowledge and problem solving ability. We find that while most recent models have near random-chance accuracy, the very largest GPT-3 model improves over random chance by almost 20 percentage points on average. However, on every one of the 57 tasks, the best models still need substantial improvements before they can reach expert-level accuracy. Models also have lopsided performance and frequently do not know when they are wrong. Worse, they still have nearrandom accuracy on some socially important subjects such as morality and law. By comprehensively evaluating the breadth and depth of a model’s academic and professional understanding, our test can be used to analyze models across many tasks and to identify important shortcomings.

authors:
M.G. Bellemare
Y. Naddaf
J. Veness
M. Bowling

title: The arcade learning environment: An evaluation platform for general agents (extended abstract)

venue: J. Artif. Intell. Res.,

authors:
Y. Bisk
R. Zellers
R.L. Bras
J. Gao
Y. Choi

title: Piqa: Reasoning about physical commonsense in natural language, 2019

authors:
T.B. Brown
B. Mann
N. Ryder
M. Subbiah
J. Kaplan
P. Dhariwal
A. Neelakantan
P. Shyam
G. Sastry
A. Askell
S. Agarwal
A. Herbert-Voss
G. Krueger
T. Henighan
R. Child
A. Ramesh
D.M. Ziegler
J. Wu
C. Winter
C. Hesse
M. Chen
E. Sigler
M. Litwin
S. Gray
B. Chess
J. Clark
C. Berner
S. McCandlish
A. Radford
I. Sutskever
D. Amodei

title: Language models are few-shot learners, 2020

authors:
P. Clark
I. Cowhey
O. Etzioni
T. Khot
A. Sabharwal
C. Schoenick
O. Tafjord

title: Think you have solved question answering? try arc, the ai2 reasoning

venue: challenge. ArXiv,

authors:
J. Devlin
M.-W. Chang
K. Lee
K. Toutanova. Bert

title: Pre-training of deep bidirectional transformers for language understanding

venue: ArXiv,

authors:
R. Geirhos
J.-H. Jacobsen
C. Michaelis
R. Zemel
W. Brendel
M. Bethge
F.A. Wichmann

title: Shortcut learning in deep neural networks, 2020

authors:
C. Guo
G. Pleiss
Y. Sun
K.Q. Weinberger

title: On calibration of modern neural networks

authors:
D. Hendrycks
M. Mazeika
T. Dietterich

title: Deep anomaly detection with outlier exposure

venue: ICLR,

authors:
D. Hendrycks
C. Burns
S. Basart
A. Critch
J. Li
D. Song
J. Steinhardt

title: Aligning ai with shared human

authors:
L. Huang
R.L. Bras
C. Bhagavatula
Y. Choi

title: Cosmos qa: Machine reading comprehension with contextual commonsense reasoning, 2019

authors:
J. Kaplan
S. McCandlish
T. Henighan
T.B. Brown
B. Chess
R. Child
S. Gray
A. Radford
J. Wu
D. Amodei

title: Scaling laws for neural language models, 2020

authors:
D. Khashabi
T. Khot
A. Sabharwal
O. Tafjord
P. Clark
H. Hajishirzi

title: Unifiedqa: Crossing format boundaries with a single qa system, 2020

authors:
T. Khot
P. Clark
M. Guerquin
P. Jansen
A. Sabharwal

title: Qasc: A dataset for question answering via sentence composition, 2019

authors:
A. Kumar
P. Liang
T. Ma

title: Verified uncertainty calibration, 2019

authors:
G. Lai
Q. Xie
H. Liu
Y. Yang
E. Hovy

title: Race: Large-scale reading comprehension dataset from examinations, 2017

authors:
Z. Lan
M. Chen
S. Goodman
K. Gimpel
P. Sharma
R. Soricut

title: Albert: A lite bert for self-supervised learning of language

venue: representations. ArXiv,

authors:
Y. Liu
M. Ott
N. Goyal
J. Du
M. Joshi
D. Chen
O. Levy
M. Lewis
L. Zettlemoyer
V. Stoyanov

title: Roberta: A robustly optimized bert pretraining approach

authors:
T. Mihaylov
P. Clark
T. Khot
A. Sabharwal

title: Can a suit of armor conduct electricity? a new dataset for open book question answering

authors:
Y. Ovadia
E. Fertig
J. Ren
Z. Nado
D. Sculley
S. Nowozin
J.V. Dillon
B. Lakshminarayanan
J. Snoek

title: Can you trust your model’s uncertainty? Evaluating predictive uncertainty under dataset shift

authors:
F. Petroni
T. Rocktäschel
P. Lewis
A. Bakhtin
Y. Wu
A.H. Miller
S. Riedel

title: Language models as knowledge bases?, 2019

authors:
A. Radford
J. Wu
R. Child
D. Luan
D. Amodei
I. Sutskever

title: Language models are unsupervised multitask learners

authors:
C. Raffel
N. Shazeer
A. Roberts
K. Lee
S. Narang
M. Matena
Y. Zhou
W. Li
P.J. Liu

title: Exploring the limits of transfer learning with a unified text-to-text transformer, 2019

authors:
M. Richardson
C.J. Burges
E. Renshaw

title: MCTest: A challenge dataset for the open-domain machine comprehension of text

venue: In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing,

authors:
A.B. Sai
A.K. Mohankumar
M.M. Khapra

title: A survey of evaluation metrics used for nlg systems

title: Computing machinery and intelligence

authors:
A. Wang
A. Singh
J. Michael
F. Hill
O. Levy
S.R. Bowman

title: Glue: A multi-task benchmark and analysis platform for natural language understanding, 2018

authors:
A. Wang
Y. Pruksachatkun
N. Nangia
A. Singh
J. Michael
F. Hill
O. Levy
S.R. Bowman

title: Superglue: A stickier benchmark for general-purpose language understanding systems, 2019

authors:
R. Zellers
A. Holtzman
Y. Bisk
A. Farhadi
Y. Choi

title: Hellaswag: Can a machine really finish your sentence

authors:
R. Zellers
A. Holtzman
E. Clark
L. Qin
A. Farhadi
Y. Choi

title: Evaluating machines by their real-world language use, 2020

heading: 1 INTRODUCTION

text:
Natural Language Processing (NLP) models have achieved superhuman performance on a number of recently proposed benchmarks. However, these models are still well below human level performance for language understanding as a whole, suggesting a disconnect between our benchmarks and the actual capabilities of these models. The General Language Understanding Evaluation benchmark (GLUE) (Wang et al., 2018) was introduced in 2018 to evaluate performance on a wide range of NLP tasks, and top models achieved superhuman performance within a year. To address the shortcomings of GLUE, researchers designed the SuperGLUE benchmark with more difficult tasks (Wang et al., 2019). About a year since the release of SuperGLUE, performance is again essentially human-level (Raffel et al., 2019). While these benchmarks evaluate linguistic skills more than overall language understanding, an array of commonsense benchmarks have been proposed to measure basic reasoning and everyday knowledge (Zellers et al., 2019; Huang et al., 2019; Bisk et al., 2019). However, these recent benchmarks have similarly seen rapid progress (Khashabi et al., 2020). Overall, the near human-level performance on these benchmarks suggests that they are not capturing important facets of language understanding.
Transformer models have driven this recent progress by pretraining on massive text corpora, including all of Wikipedia, thousands of books, and numerous websites. These models consequently see extensive information about specialized topics, most of which is not assessed by existing NLP benchmarks. It consequently remains an open question just how capable current language models are at learning and applying knowledge from many domains.
To bridge the gap between the wide-ranging knowledge that models see during pretraining and the existing measures of success, we introduce a new benchmark for assessing models across a diverse set of subjects that humans learn. We design the benchmark to measure knowledge acquired during pretraining by evaluating models exclusively in zero-shot and few-shot settings. This makes the benchmark more challenging and more similar to how we evaluate humans. The benchmark covers 57 subjects across STEM, the humanities, the social sciences, and more. It ranges in difficulty from an elementary level to an advanced professional level, and it tests both world knowledge and problem solving ability. Subjects range from traditional areas, such as mathematics and history, to more
Few Shot Prompt and Predicted Answer

heading: How many numbers are in the list 25, 26, ..., 100?

text: (A) 75 (B) 76 (C) 22 (D) 23

heading: Answer: B

text: Compute i + i2 + i3 + ··· + i 258 + i259. (A) -1 (B) 1 (C) i (D) -i

heading: Answer: A

text:
If 4 daps = 7 yaps, and 5 yaps = 3 baps, how many daps equal 42 baps? (A) 28 (B) 21 (C) 40 (D) 30 Answer: C␣
The following are multiple choice questions about high school mathematics.
(a) An example of few-shot learning and inference using GPT-3. The blue underlined bold text is the autocompleted response from GPT-3, while the preceding text is the user-inputted prompt. In this 2-shot learning example, there are two instruction examples and one initially incomplete example. On average, GPT-3 has low accuracy on high school mathematics questions.
Small Medium Large X-Large Model Size
20
30
40
50
60
70
80
90
Pe rfo
rm an
ce (%
)
GPT-3 Few Shot Test Performance Commonsense Linguistics Knowledge (Ours)
(b) Performance on a commonsense benchmark (HellaSwag), a linguistic understanding benchmark (SuperGLUE), and the massive multitask test. On previous benchmarks, smaller models start well above random chance levels and exhibit more continuous improvements with model size increases, but on our test, GPT-3 moves beyond random chance with the largest model.
specialized areas like law and ethics (Hendrycks et al., 2020). The granularity and breadth of the subjects makes the benchmark ideal for identifying a model’s blind spots.
We find that meaningful progress on our benchmark has only become possible in recent months. In particular, few-shot models up to 13 billion parameters (Brown et al., 2020) achieve random chance performance of 25% accuracy, but the 175 billion parameter GPT-3 model reaches a much higher 43.9% accuracy (see Figure 1b). On the other hand, unlike human professionals GPT-3 does not excel at any single subject. Instead, we find that performance is lopsided, with GPT-3 having almost 70% accuracy for its best subject but near-random performance for several other subjects.
Our results indicate that while recent advances have been impressive, state-of-the-art models still struggle at learning and applying knowledge from pretraining. The tasks with near-random accuracy include calculation-heavy subjects such as physics and mathematics and subjects related to human values such as law and morality. This second weakness is particularly concerning because it will be important for future models to have a strong understanding of what is legal and what is ethical. Worryingly, we also find that GPT-3 does not have an accurate sense of what it does or does not know since its average confidence can be up to 24% off from its actual accuracy. We comprehensively evaluate the breadth and depth of a model’s text understanding by covering numerous topics that humans are incentivized to learn. Since our test consists in 57 tasks, it can be used to analyze aggregate properties of models across tasks and to track important shortcomings. The test and code is available at github.com/hendrycks/test.

heading: 2 RELATED WORK

text:
Pretraining. The dominant paradigm in NLP is to pretrain large models on massive text corpora including educational books and websites. In the process, these models are exposed to information about a wide range of topics. Petroni et al. (2019) found that recent models learn enough information from pretraining that they can serve as knowledge bases. However, no prior work has comprehensively measured the knowledge models have across many real-world domains.
Until recently, researchers primarily used fine-tuned models on downstream tasks (Devlin et al., 2019). However, larger pretrained models like GPT-3 (Brown et al., 2020) have made it possible to achieve competitive performance without fine-tuning by using few-shot learning, which removes the need for a large fine-tuning set. With the advent of strong zero-shot and few-shot learning, it is now possible to curate a diverse set of tasks for evaluation and remove the possibility of models on “spurious cues” (Geirhos et al., 2020; Hendrycks et al., 2019b) in a dataset to achieve high performance.
Benchmarks. Many recent benchmarks aim to assess a model’s general world knowledge and basic reasoning ability by testing its “commonsense.” A number of commonsense benchmarks have been
proposed in the past year, but recent models are already nearing human-level performance on several of these, including HellaSwag (Zellers et al., 2019), Physical IQA (Bisk et al., 2019), and CosmosQA (Huang et al., 2019). By design, these datasets assess abilities that almost every child has. In contrast, we include harder specialized subjects that people must study to learn.
Some researchers have suggested that the future of NLP evaluation should focus on Natural Language Generation (NLG) (Zellers et al., 2020), an idea that reaches back to the Turing Test (Turing, 1950). However, NLG is notoriously difficult to evaluate and lacks a standard metric (Sai et al., 2020). Consequently, we instead create a simple-to-evaluate test that measures classification accuracy on multiple choice questions.
While several question answering benchmarks exist, they are comparatively limited in scope. Most either cover easy topics like grade school subjects for which models can already achieve strong performance (Clark et al., 2018; Khot et al., 2019; Mihaylov et al., 2018; Clark et al., 2019), or are focused on linguistic understanding in the form of reading comprehension (Lai et al., 2017; Richardson et al., 2013). In contrast, we include a wide range of difficult subjects that go far beyond linguistic understanding.

heading: 3 A MULTITASK TEST

text:
We create a massive multitask test consisting of multiple-choice questions from various branches of knowledge. The test spans subjects in the humanities, social sciences, hard sciences, and other areas that are important for some people to learn. There are 57 tasks in total, which is also the number of Atari games (Bellemare et al., 2013), all of which are listed in Appendix B. The questions in the dataset were manually collected by graduate and undergraduate students from freely available sources online. These include practice questions for tests such as the Graduate Record Examination and the United States Medical Licensing Examination. It also includes questions designed for undergraduate courses and questions designed for readers of Oxford University Press books. Some tasks cover a subject, like psychology, but at a specific level of difficulty, such as “Elementary,” “High School,” “College,” or “Professional.” For example, the “Professional Psychology” task draws on questions from freely available practice questions for the Examination for Professional Practice in Psychology, while the “High School Psychology” task has questions like those from Advanced Placement Psychology examinations.
We collected 15908 questions in total, which we split into a few-shot development set, a validation set, and a test set. The few-shot development set has 5 questions per subject, the validation set may be used for selecting hyperparameters and is made of 1540 questions, and the test set has 14079 questions. Each subject contains 100 test examples at the minimum, which is longer than most exams designed to assess people.
Human-level accuracy on this test varies. Unspecialized humans from Amazon Mechanical Turk obtain 34.5% accuracy on this test. Meanwhile, expert-level performance can be far higher. For example, real-world test-taker human accuracy at the 95th percentile is around 87% for US Medical Licensing Examinations, and these questions make up our “Professional Medicine” task. If we take the 95th percentile human test-taker accuracy for exams that build up our test, and if we make an educated guess when such information is unavailable, we then estimate that expert-level accuracy is approximately 89.8%.
Since our test aggregates different subjects and several levels of difficulty, we measure more than straightforward commonsense or narrow linguistic understanding. Instead, we measure arbitrary
When you drop a ball from rest it accelerates downward at 9.8 m/s². If you instead throw it downward assuming no air resistance its acceleration immediately after leaving your hand is (A) 9.8 m/s² (B) more than 9.8 m/s² (C) less than 9.8 m/s² (D) Cannot say unless the speed of throw is given. C on ce pt ua l Ph ys ic s
C ol
le ge
M at
he m
at ic s In the complex z-plane, the set of points satisfying the equation z² = |z|² is a (A) pair of points (B) circle (C) half-line (D) line
Figure 4: Examples from the Conceptual Physics and College Mathematics STEM tasks.
real-world text understanding. Since models are pretrained on the Internet, this enables us to test how well they can extract useful knowledge from massive corpora. Future models that use this test could be single models or a mixture of experts model. To succeed at our test, future models should be well-rounded, possess extensive world knowledge, and develop expert-level problem solving ability. These properties make the test likely to be an enduring and informative goalpost.

heading: 3.1 HUMANITIES

text: The humanities is a group of disciplines that make use of qualitative analysis and analytic methods rather than scientific empirical methods. Branches of the humanities include law, philosophy, history, and so on (Appendix B). Mastering these subjects requires a variety of skills. For example, legal understanding requires knowledge of how to apply rules and standards to complex scenarios, and also provide answers with stipulations and explanations. We illustrate this in Figure 2. Legal understanding is also necessary for understanding and following rules and regulations, a necessary capability to constrain open-world machine learning models. For philosophy, our questions cover concepts like logical fallacies, formal logic, and famous philosophical arguments. It also covers moral scenarios, including questions from the ETHICS dataset (Hendrycks et al., 2020) that test a model’s understanding of normative statements through predicting widespread moral intuitions about diverse everyday scenarios. Finally, our history questions cover a wide range of time periods and geographical locations, including prehistory and other advanced subjects.

heading: 3.2 SOCIAL SCIENCE

text: Social science includes branches of knowledge that examine human behavior and society. Subject areas include economics, sociology, politics, geography, psychology, and so on. See Figure 3 for an example question. Our economics questions include microeconomics, macroeconomics, and econometrics, and cover different types of problems, including questions that require a mixture of world knowledge, qualitative reasoning, or quantitative reasoning. We also include important but more esoteric topics such as security studies in order to test the boundaries of what is experienced and learned during pretraining. Social science also includes psychology, a field that may be especially important for attaining a nuanced understanding of humans.

heading: 3.3 SCIENCE, TECHNOLOGY, ENGINEERING, AND MATHEMATICS (STEM)

text:
STEM subjects include physics, computer science, mathematics, and more. Two examples are shown in Figure 4. Conceptual physics tests understanding of simple physics principles and may be thought
of as a harder version of the physical commonsense benchmark Physical IQA (Bisk et al., 2019). We also test mathematical problem solving ability at various levels of difficulty, from the elementary to the college level. College mathematics questions, like those found on the GRE mathematics subject test, often require chains of reasoning and abstract knowledge. To encode mathematics expressions, we use LaTeX or symbols such as * and ˆ for multiplication and exponentiation respectively. STEM subjects require knowledge of empirical methods, fluid intelligence, and procedural knowledge.

heading: 3.4 OTHER

text: There is a long tail of subjects that either do not neatly fit into any of the three preceding categories or for which there are not thousands of freely available questions. We put these subjects into Other. This section includes the Professional Medicine task, which has difficult questions that require humans many years of study to master. An example is depicted in Figure 5. This section also contains business topics like finance, accounting, and marketing, as well as knowledge of global facts. The latter includes statistics about poverty in different countries over time, which may be necessary for having an accurate model of the world internationally.

heading: 4 EXPERIMENTS

heading: 4.1 SETUP

text:
Assessment and Models. To measure performance on our multitask test, we compute the classification accuracy across all examples and tasks. We evaluate GPT-3 (Brown et al., 2020) and UnifiedQA (Khashabi et al., 2020). For GPT-3 we use the OpenAI API, which provides access to four model variants, “Ada,” “Babbage,” “Curie,” and “Davinci,” which we refer to as “Small” (2.7 billion parameters), “Medium” (6.7 billion), “Large” (13 billion) and “X-Large” (175 billion). UnifiedQA uses the T5 (Raffel et al., 2019) text-to-text backbone and is fine-tuned on previously proposed question answering datasets (Lai et al., 2017), where the prediction is the class with the highest token overlap with UnifiedQA’s text output. Since UnifiedQA is fine-tuned on other datasets, we evaluate it without any further tuning to assess its transfer accuracy. We also fine-tune RoBERTa-base, ALBERT-xxlarge, and GPT-2 on UnifiedQA training data and our dev+val set. We primarily focus on UnifiedQA and GPT-3 in the rest of this document, but additional discussion of RoBERTa, ALBERT, and GPT-2 is in Appendix A.
Few-Shot Prompt. We feed GPT-3 prompts like that shown in Figure 1a. We begin each prompt with “The following are multiple choice questions (with answers) about [subject].” For zero-shot evaluation, we append the question to the prompt. For few-shot evaluation, we add up to 5 demonstration examples with answers to the prompt before appending the question. All prompts end with “Answer: ”. The model then produces probabilities for the tokens “A,” “B,” “C,” and “D,” and we treat the highest probability option as the prediction. For consistent evaluation, we create a dev set with 5 fixed few-shot examples for each subject.

heading: 4.2 RESULTS

text:
Model Size and Accuracy. We compare the few-shot accuracy of each GPT-3 size in Table 1. We find that the three smaller GPT-3 models have near random accuracy (around 25%). In contrast, we find that the X-Large 175 billion parameter GPT-3 model performs substantially better than random, with an accuracy of 43.9%. We also find qualitatively similar results in the zero-shot setting. While the smaller models have around 25% zero-shot accuracy, Figure 10 in Appendix A shows that the largest GPT-3 model has a much higher zero-shot accuracy of about 37.7%. Brown et al. (2020) also observe that larger GPT-3 models perform better, though progress tends to be steadier. In Figure 1b we show that non-random accuracy on the multitask test emerged with recent large few-shot models compared to datasets that assess commonsense and linguistic understanding.
To test the usefulness of fine-tuning instead of few-shot learning, we also evaluate UnifiedQA models. UnifiedQA has the advantage of being fine-tuned on other question answering datasets, unlike GPT-3. We assess UnifiedQA by evaluating its transfer performance without any additional fine-tuning. The largest UnifiedQA model we test has 11 billion parameters, which is slightly smaller than GPT-3 Large. Nevertheless, we show in Table 1 that it attains 48.9%
accuracy. This performs better than the few-shot GPT-3 X-Large model, despite UnifiedQA have an order of magnitude fewer parameters. We also find that even the smallest UnifiedQA variant, with just 60 million parameters, has approximately 29.3% accuracy. These results suggest that while model size is a key component for achieving strong performance, fine-tuning also helps.
Comparing Disciplines. Using our test, we discover that GPT-3 and UnifiedQA have lopsided performance and several substantial knowledge gaps. Figure 6 shows the accuracy of GPT-3 (fewshot) and UnifiedQA for all 57 tasks. It shows the both models are below expert-level performance for all tasks, with GPT-3’s accuracy ranging from 69% for US Foreign Policy to 26% for College Chemistry. UnifiedQA does best on marketing, with an accuracy of 82.5%.
Overall, models do poorly on highly procedural problems. Figure 6 shows that calculation-heavy STEM subjects tend to have low accuracy compared to verbal subjects. For GPT-3, 9 out of the 10
Figure 7: GPT-3’s completion for two prompts testing knowledge of the order of operations. The blue underlined bold text is the autocompleted response from GPT-3. While it knows about the order of operations, it sometimes does not know how to apply its knowledge.
lowest-accuracy tasks are STEM subjects that emphasize mathematics or calculations. We speculate that is in part because GPT-3 acquires declarative knowledge more readily than procedural knowledge. For example, many questions in Elementary Mathematics require applying the order of operations for arithmetic, which is described by the acronym PEMDAS (Parentheses Exponents Multiplication Division Addition Subtraction). In Figure 7, we confirm that GPT-3 is aware of the acronym PEMDAS. However, it does not consistently apply PEMDAS to actual problems. On the other hand, procedural understanding is not its only weak point. We find that some verbal tasks such as Moral Scenarios from Hendrycks et al. (2020) and Professional Law also have especially low accuracy.
Our test also shows that GPT-3 acquires knowledge quite unlike humans. For example, GPT-3 learns about topics in a pedagogically unusual order. GPT-3 does better on College Medicine (47.4%) and College Mathematics (35.0%) than calculation-heavy Elementary Mathematics (29.9%). GPT-3 demonstrates unusual breadth, but it does not master a single subject. Meanhwhile we suspect humans have mastery in several subjects but not as much breadth. In this way, our test shows that GPT-3 has many knowledge blindspots and has capabilities that are lopsided.
Calibration. We should not trust a model’s prediction unless the model is calibrated, meaning that its confidence is a good estimate of the actual probability the prediction is correct. However, large neural networks are often miscalibrated (Guo et al., 2017), especially under distribution shift (Ovadia et al., 2019). We evaluate the calibration of GPT-3 by testing how well its average confidence estimates its actual accuracy for each subject. We show the results in Figure 8, which demonstrates that GPT-3 is uncalibrated. In fact, its confidence is only weakly related to its actual accuracy in the zero-shot setting, with the difference between its accuracy and confidence reaching up to 24% for some subjects. Another calibration measure is the Root Mean Squared (RMS) calibration error (Hendrycks et al., 2019a; Kumar et al., 2019). Many tasks have miscalibrated predictions, such as Elementary Mathematics which has a zero-shot RMS calibration error of 19.4%. Models are only somewhat more calibrated in the few-shot setting, as shown in Appendix A. These results suggest that model calibration has wide room for improvement.

heading: 5 DISCUSSION

text:
Multimodal Understanding. While text is capable of conveying an enormous number of concepts about the world, many important concepts are conveyed mainly through other modalities, such as images, audio, and physical interaction (Bisk et al., 2020). Existing large-scale NLP models, such as GPT-3, do not incorporate multimodal information, so we design our benchmark to capture a diverse array of tasks in a text-only format. However, as models gain the ability to process multimodal inputs, benchmarks should be designed to reflect this change. One such benchmark could be a “Turk Test,” consisting of Amazon Mechanical Turk Human Intelligence Tasks. These are well-defined tasks that require models to interact with flexible formats and demonstrate multimodal understanding.
The Internet as a Training Set. A major distinction between our benchmark and previous multitask NLP benchmarks is that we do not require large training sets. Instead, we assume that models have acquired the requisite knowledge from reading vast quantities of diverse text from the Internet. This
process is typically called pretraining, but it can be thought of as training in its own right, where the downstream evaluation is demonstrating whatever knowledge we would expect a human to pick up from reading the same text.
This motivates us to propose a methodological change so that models are trained more like how humans learn. While most previous machine learning benchmarks have models learn from a large question bank, humans primarily learn new subjects by reading books and listening to others talk about the topic. For specialized subjects such as Professional Law, massive legal corpora are available, such as the 164-volume legal encyclopedia Corpus Juris Secundum, but there are fewer than 5,000 multistate bar exam questions available. Learning the entire law exclusively through a small number of practice tests is implausible, so future models must learn more during pretraining.
For this reason we assess pretrained models in a zero-shot, few-shot, or transfer setting and we provide a dev, val, and test set for each task. The dev set is used for few-shot prompts, the val set could be used for hyperparameter tuning, and the test set is used to compute the final accuracy. Importantly, the format of our evaluation is not identical to the format in which information is acquired during pretraining. This has the benefit of obviating concerns about spurious training set annotation artifacts (Geirhos et al., 2020; Hendrycks et al., 2019b) and is in stark contrast to the previous paradigm of identically distributed training and test sets. This change also enables collecting a much more extensive and diverse set of tasks for evaluation. We anticipate our methodology becoming more widespread as models improve at extracting information from diverse online sources.
Model Limitations. We find that current large-scale Transformers have wide room for improvement. They are notably poor at modeling human (dis)approval, as evident by the low performance on the Professional Law and Moral Scenarios tasks. For future systems to be aligned with human values, high performance on these tasks is crucial (Hendrycks et al., 2020), so future research should especially aim to increase accuracy on these tasks. Models also have difficulty performing calculations, so much so that they exhibit poor performance on Elementary Mathematics and many other STEM subjects with “plug and chug” problems. Additionally, they do not match expert-level performance (90%) on any subject, so for all subjects it is subhuman. On average, models are only now starting to move beyond random-chance accuracy levels.
Addressing these shortcomings may be challenging. To illustrate this, we attempted to create a better Professional Law model by pretraining on specialized data but achieved only limited success. We collected approximately 2,000 additional Professional Law training examples. After fine-tuning a RoBERTa-base model (Liu et al., 2019) using this custom training set, our model attained 32.8% test accuracy. To test the impact of additional specialized training data, we also had RoBERTa continue pretraining on approximately 1.6 million legal case summaries using Harvard’s Law Library case law corpus case.law, but after fine-tuning it only attained 36.1% accuracy. This suggests that while additional pretraining on relevant high quality text can help, it may not be enough to substantially increase the performance of current models.
It is unclear whether simply scaling up existing language models will solve the test. Current understanding indicates that a 10× increase in model size must be accompanied by an approximate 5× increase in data (Kaplan et al., 2020). Aside from the tremendous expense in creating multi-trillion parameter language models, data may also become a bottleneck, as there is far less written about esoteric branches of knowledge than about everyday situations.

heading: 6 CONCLUSION

text: We introduced a new test that measures how well text models can learn and apply knowledge encountered during pretraining. By covering 57 subjects at varying levels of difficulty, the test assesses language understanding in greater breadth and depth than previous benchmarks. We found that it has recently become possible for models to make meaningful progress on the test, but that state-of-the-art models have lopsided performance and rarely excel at any individual task. We also showed that current models are uncalibrated and have difficulty with tasks that require calculations. Worryingly, models also perform especially poorly on socially relevant subjects including morality and law. Our expansive test can help researchers pinpoint important shortcomings of models, making it easier to gain a clearer picture of state-of-the-art capabilities.

heading: ACKNOWLEDGEMENTS

text: We would like to thank the following for their helpful comments: Oyvind Tafjord, Jan Leike, David Krueger, Alex Tamkin, Girish Sastry, and Henry Zhu. DH is supported by the NSF GRFP Fellowship and an Open Philanthropy Project Fellowship. This research was also supported by the NSF Frontier Award 1804794.

heading: A ADDITIONAL ANALYSIS

text:
This appendix includes figures with sorted results (Figure 9), few-shot examples vs. accuracy (Figure 10), and few-shot calibration (Figure 11). It also includes sections on fine-tuning, error analysis, and format sensitivity.
0 20 40 60 80 100
Accuracy (%)
College Chemistry Moral Scenarios College Physics
High School Physics High School Mathematics
Formal Logic Elementary Mathematics
Abstract Algebra High School Statistics
Machine Learning Econometrics
High School Chemistry Professional Accounting
Professional Law College Mathematics
Professional Medicine Conceptual Physics
Global Facts High School Comp Sci
Medical Genetics High School Macroeconomics High School Microeconomics
Moral Disputes Professional Psychology
College Biology Virology
College Comp Sci Business Ethics
Nutrition College Medicine
Anatomy Clinical Knowledge
Logical Fallacies High School Biology
Public Relations Astronomy
Electrical Engineering Human Aging
Philosophy Security Studies
Prehistory High School US History
Sociology High School European History
Human Sexuality Jurisprudence
World Religions International Law
High School World History Management
Computer Security High School Geography
High School Gov't and Politics Marketing
Miscellaneous High School Psychology
US Foreign Policy GPT-3 Results
Random Chance
0 10 20 30 40 50 60 70 80 90 100
Accuracy (%)
Moral Scenarios Formal Logic
Abstract Algebra Econometrics
High School Mathematics College Physics
Machine Learning High School Statistics
College Chemistry Elementary Mathematics
College Mathematics High School Chemistry
Global Facts Professional Law Medical Genetics
Professional Accounting College Biology
High School Physics Anatomy
College Comp Sci Conceptual Physics
College Medicine Virology
Professional Medicine Astronomy
High School Macroeconomics Electrical Engineering
Professional Psychology Security Studies
Human Sexuality Nutrition
High School Comp Sci Prehistory
Clinical Knowledge High School Biology
Human Aging High School Microeconomics
Philosophy Public Relations World Religions Moral Disputes
Logical Fallacies High School European History
High School US History High School World History
Miscellaneous US Foreign Policy
Computer Security Sociology
International Law High School Geography
Jurisprudence Business Ethics
High School Psychology Management
High School Gov't and Politics Marketing
UnifiedQA Results
Random Chance
Figure 9: On the left are GPT-3 few shot accuracies for all of the 57 tasks. On the right are UnifiedQA transfer accuracies for all of the 57 tasks. For both models, capabilities are lopsided.

heading: A.1 ANALYSIS WITH MORE FINE-TUNED MODELS

text:
We primarily analyzed models with more than 10 billion parameters in the main body of the paper. For this section, we analyze smaller models including RoBERTa-base (125 million parameters) (Liu
et al., 2019), ALBERT-xxlarge (223 million parameters) (Lan et al., 2020), and GPT-2 (1,558 million parameters) (Radford et al., 2019). Models are fine-tuned to predict one of four classes using the UnifiedQA MCQ questions and using our dev+val set. We test on our multitask test set.
We observe that these smaller models can attain better-than-random accuracy. RoBERTa-base attains an overall accuracy of 27.9%, with 27.9% accuracy for the humanities, 28.8% for social sciences, 27.0% for STEM, and 27.7% for other. ALBERT-xxlarge attains an accuracy of 27.1%, with 27.2% accuracy for the humanities, 25.7% for the social sciences, 27.7% for STEM, and 27.9% for other. GPT-2 attains an accuracy of 32.4%, with 32.8% accuracy for the humanities, 33.3% for the social sciences, 30.2% for STEM, and 33.1% for other.
Compare this to UnifiedQA’s smallest variant, which has just 60 million parameters and approximately 29.3% accuracy. It obtains higher accuracy than RoBERTa and ALBERT, even though it has fewer parameters. This suggests that its larger pretraining dataset enables higher accuracy. Likewise, UnifiedQA with 3 billion parameters attains 43.7%, while the similarly sized GPT-2 model with 1.5 billion parameters attains 32.4% accuracy. This again suggests that T5’s larger pretraining dataset size (and therefore UnifiedQA’s pretraining dataset size) can increase accuracy.

heading: A.2 ERROR ANALYSIS

text: We qualitatively analyze when GPT-3 makes high confidence mistakes. We find that while many of these mistakes were clearly wrong, many were mistakes that a human might make. For example, one question it got wrong was “How many chromosomes do all human somatic cells contain?” The correct answer is 46, while few-shot GPT-3 predicted 23 with confidence 97.5%. This answer would have been correct if the question asked about the number of pairs of chromosomes. Similarly, many of its other high confidence mistakes were also correct answers to slightly different questions.

heading: A.3 FORMAT SENSITIVITY

text:
While different question formatting choices often lead to similar GPT-3 accuracies, we find that UnifiedQA is more sensitive. UnifiedQA’s input format is of the form
QUESTION1 \\n (A) CHOICE1 (B) CHOICE2 (C) CHOICE3 (D) CHOICE4</s>
where questions and choices are normalized and made lowercase. If we remove the </s> from the input, accuracy declines by several percentage points.

heading: B TEST DETAILS

heading: B.1 TASK DESCRIPTIONS AND EXAMPLES

text: We provide analysis of question length and difficulty in Figure 12. We list all tasks and the topics they test in Table 2. We also provide an example for each task starting with Figure 14.

heading: B.2 EXACT QUESTION AND ANSWER CONTAMINATION

text:
Since language models train on vast text corpora, there is some chance that they have seen the exact question and answer during pretraining. If they memorized the exact question and answer, then they would attain higher accuracy than their true ability. Likewise, a question’s entropy would be especially low if it were memorized. Memorized questions and answers should have low entropy and
high accuracy. However, in Figure 13, we see that accuracy and question entropy are not positively correlated, suggesting that the test’s low-entropy questions do not correspond to memorized (and thereby correctly predicted) answers. This suggests that our exact questions were not memorized. However, during pretraining models encountered text related to our questions through processing Wikipedia. We also note that most of our questions came from PDFs or websites where questions and answers are on separate pages.
See Brown et al. (2020) for a previous discussion of contamination showing that the phenomena hardly affects performance. To reduce the probability that future models encounter exact questions during test-time, we will provide a list of question sources.

title: MEASURING MASSIVE MULTITASK LANGUAGE UNDERSTANDING


summaries: This work explores a pretraining strategy (similar to https://arxiv.org/abs/1606.03622) to the problem of table question answering. More specifically a synchronous context-free grammar (SCFG) is first learned from training data (with manual alignment of entities/phrases). Then the SCFG is used to generate more full supervision data for Roberta model pretraining. The training objective is a combination of two parts: SQL Semantic Precision (SSP) predicts elements in SQL given the question on the synthetic data, and masked-language modeling (MLM) on the natural (training) data.

summaries: The paper provides a interesting direction in pre-training for table semantic parsing. In particular, it proposes to first collect a collection of pseudo question-SQL pairs in an automatic way, based on tables from WikiTables tables and tables and databases in the training sets  SPIDER and WIKISQL. After that, masked language modeling and a newly introduced task called SSP, which is, given a natural language sentence and table headers, to predict whether a column appears in the SQL query and what operation is triggered. Experiments on Spider and WikiSQL show that the model achieves new state-of-the-art.

summaries: This paper presents a general-purpose pre-training approach for jointly encoding utterances and relational tables in the task of table semantic parsing, where a natural language utterance is transduced into an executable query (e.g., SQL) over relational database tables. A core challenge in table semantic parsing is to understand the compositional semantics in utterances, and further ground salient entities and relations in the utterance onto the corresponding tabular schema (e.g., columns, cells). To improve understanding and grounding of compositional utterances, the authors propose fine-tuning a pre-trained masked language model (RoBERTa) using linearized table headers paired with synthetic utterances generated from a synchronous context free grammar, via an objective that encourages the model to discover the syntactic roles of columns mentioned in the input utterance. Experiments over four datasets demonstrated strong results when the this newly proposed pre-training objective is combined with classical masked language modeling objective.

abstractText: We present GRAPPA, an effective pre-training approach for table semantic parsing that learns a compositional inductive bias in the joint representations of textual and tabular data. We construct synthetic question-SQL pairs over high-quality tables via a synchronous context-free grammar (SCFG). We pre-train GRAPPA on the synthetic data to inject important structural properties commonly found in table semantic parsing into the pre-training language model. To maintain the model’s ability to represent real-world data, we also include masked language modeling (MLM) on several existing table-and-language datasets to regularize our pre-training process. Our proposed pre-training strategy is much data-efficient. When incorporated with strong base semantic parsers, GRAPPA achieves new state-of-the-art results on four popular fully supervised and weakly supervised table semantic parsing tasks. The pre-trained embeddings can be downloaded at https://huggingface.co/Salesforce/grappa_large_jnt.

authors:
Rishabh Agarwal
Chen Liang
Dale Schuurmans
Mohammad Norouzi

title: Learning to generalize from sparse and underspecified rewards

venue: In ICML,

title: Good-enough compositional data augmentation

venue: Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 7556–7566,

authors:
Yoav Artzi
Luke Zettlemoyer

title: Weakly supervised learning of semantic parsers for mapping instructions to actions

venue: Transactions of the Association forComputational Linguistics,

authors:
Jonathan Berant
Percy Liang

title: Semantic parsing via paraphrasing. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)

authors:
Chandra Bhagavatula
Thanapon Noraset
Doug Downey

title: Tabel: Entity linking in web tables

venue: In International Semantic Web Conference,

authors:
Ben Bogin
Matt Gardner
Jonathan Berant

title: Global reasoning over database structures for text-to-sql

venue: parsing. ArXiv,

authors:
Giovanni Campagna
Agata Foryciarz
Mehrad Moradshahi
Monica S. Lam

title: Zero-shot transfer learning with synthesized data for multi-domain dialogue state tracking

venue: In Proceedings of 58th Annual Meeting of the Association for Computational Linguistics

authors:
Wenhu Chen
Hongmin Wang
Jianshu Chen
Yunkai Zhang
Hong Wang
Shiyang Li
Xiyou Zhou
William Yang Wang

title: Tabfact: A large-scale dataset for table-based fact verification

authors:
Wenhu Chen
Hanwen Zha
Zhiyu Chen
Wenhan Xiong
Hong Wang
William Wang

title: Hybridqa: A dataset of multi-hop question answering over tabular and textual data

authors:
Donghyun Choi
Myeong Cheol Shin
Eunggyun Kim
Dong Ryeol Shin

title: Ryansql: Recursively applying sketch-based slot fillings for complex text-to-sql in cross-domain databases

authors:
Pradeep Dasigi
Matt Gardner
Shikhar Murty
Luke S. Zettlemoyer
Eduard H. Hovy

title: Iterative search for weakly supervised semantic parsing

authors:
Jacob Devlin
Ming-Wei Chang
Kenton Lee
Kristina Toutanova. Bert

title: Pre-training of deep bidirectional transformers for language understanding

venue: In NAACL-HLT,

authors:
Li Dong
Mirella Lapata

title: Coarse-to-fine decoding for neural semantic parsing. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 731–742

venue: Association for Computational Linguistics,

authors:
Catherine Finegan-Dollak
Jonathan K. Kummerfeld
Li Zhang
Sesh Sadasivam
Rui Zhang
Dragomir Radev

title: Improving text-to-sql evaluation methodology. In ACL 2018

authors:
Jiaqi Guo
Zecheng Zhan
Yan Gao
Yan Xiao
Jian-Guang Lou
Ting Liu
Dongmei Zhang

title: Towards complex text-to-sql in cross-domain database with intermediate representation

authors:
Tong Guo
Huilin Gao

title: Content enhanced bert-based text-to-sql generation

venue: Technical report,

authors:
Kelvin Guu
Kenton Lee
Zora Tung
Panupong Pasupat
Ming-Wei Chang

title: Realm: Retrievalaugmented language model pre-training

authors:
Pengcheng He
Yi Mao
Kaushik Chakrabarti
Weizhu Chen

title: X-sql: reinforce schema representation with context

authors:
Dan Hendrycks
Kevin Gimpel

title: A baseline for detecting misclassified and out-of-distribution examples in neural networks. ArXiv

authors:
Jonathan Herzig
Jonathan Berant

title: Decoupling structure and lexicon for zero-shot semantic parsing

venue: In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing,

authors:
Jonathan Herzig
P. Nowak
Thomas Müller
Francesco Piccinno
Julian Martin Eisenschlos

title: Tapas: Weakly supervised table parsing via pre-training

venue: In ACL,

authors:
Jonathan Herzig
Paweł Krzysztof Nowak
Thomas Müller
Francesco Piccinno
Julian Martin Eisenschlos

venue: arXiv preprint arXiv:2004.02349,

authors:
Wonseok Hwang
Jinyeung Yim
Seunghyun Park
Minjoon Seo

title: A comprehensive exploration on wikisql with table-aware word contextualization

authors:
Srinivasan Iyer
Ioannis Konstas
Alvin Cheung
Jayant Krishnamurthy
Luke Zettlemoyer

title: Learning a neural semantic parser from user

venue: feedback. CoRR,

authors:
Robin Jia
Percy Liang

title: Data recombination for neural semantic parsing

venue: In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume

authors:
Diederik P. Kingma
Jimmy Ba

title: Adam: A method for stochastic optimization

authors:
Mike Lewis
Marjan Ghazvininejad
Gargi Ghosh
Armen Aghajanyan
Sida Wang
Luke Zettlemoyer

title: Pre-training via paraphrasing

venue: arXiv preprint arXiv:2006.15020,

authors:
Mike Lewis
Yinhan Liu
Naman Goyal
Marjan Ghazvininejad
Abdelrahman Mohamed
Omer Levy
Veselin Stoyanov
Luke Zettlemoyer

title: BART: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension

authors:
Fei Li
HV Jagadish

title: Constructing an interactive natural language interface for relational databases

authors:
Chen Liang
Mohammad Norouzi
Jonathan Berant
Quoc V. Le
Ni Lao

title: Memory augmented policy optimization for program synthesis and semantic parsing

venue: NeurIPS,

authors:
Xi Victoria Lin
Richard Socher
Caiming Xiong

title: Bridging textual and tabular data for crossdomain text-to-sql semantic parsing

venue: In Findings of the 2020 Conference on Empirical Methods in Natural Language Processing, EMNLP Findings

authors:
Yinhan Liu
Myle Ott
Naman Goyal
Jingfei Du
Mandar Joshi
Danqi Chen
Omer Levy
Mike Lewis
Luke S. Zettlemoyer
Veselin Stoyanov

authors:
Jiasen Lu
Dhruv Batra
Devi Parikh
Stefan Lee

title: Vilbert: Pretraining task-agnostic visiolinguistic representations for vision-and-language tasks

venue: Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems

authors:
Qin Lyu
Kaushik Chakrabarti
Shobhit Hathi
Souvik Kundu
Jianwen Zhang
Zheng Chen

title: Hybrid ranking network for text-to-sql

venue: Technical Report MSR-TR-2020-7, Microsoft Dynamics

authors:
Sewon Min
Danqi Chen
Hannaneh Hajishirzi
Luke Zettlemoyer

title: A discrete hard em approach for weakly supervised question answering

authors:
Ankur P Parikh
Xuezhi Wang
Sebastian Gehrmann
Manaal Faruqui
Bhuwan Dhingra
Diyi Yang
Dipanjan Das

title: Totto: A controlled table-to-text generation dataset

venue: arXiv preprint arXiv:2004.14373,

authors:
Panupong Pasupat
Percy Liang

title: Compositional semantic parsing on semi-structured tables. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing of the Asian Federation of Natural Language Processing, ACL 2015

authors:
Jeffrey Pennington
Richard Socher
Christopher D. Manning

title: Glove: Global vectors for word representation

venue: In EMNLP,

authors:
Matthew E. Peters
Mark Neumann
Robert Logan
Roy Schwartz
Vidur Joshi
Sameer Singh
Noah A. Smith

title: Knowledge enhanced contextual word representations

venue: In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP),

authors:
Dragomir Radev
Rui Zhang
Amrit Rau
Abhinand Sivaprasad
Chiachun Hsieh
Nazneen Fatema Rajani
Xiangru Tang
Aadit Vyas
Neha Verma
Pranav Krishna
Yangxiaokang Liu
Nadia Irwanto
Jessica Pan
Faiaz Rahman
Ahmad Zaidi
Murori Mutuma
Yasin Tarabar
Ankit Gupta
Tao Yu
Yi Chern Tan
Xi Victoria Lin
Caiming Xiong
Richard Socher

title: Dart: Open-domain structured data record to text generation

venue: arXiv preprint arXiv:2007.02871,

authors:
Tianze Shi
Kedar Tatwawadi
Kaushik Chakrabarti
Yi Mao
Oleksandr Polozov
Weizhu Chen

title: Incsql: Training incremental text-to-sql parsers with non-deterministic oracles

venue: arXiv preprint arXiv:1809.05054,

authors:
Bailin Wang
Ivan Titov
Mirella Lapata

title: Learning semantic parsers from denotations with latent structured alignments and abstract programs

venue: In Proceedings of EMNLP,

authors:
Bailin Wang
Richard Shin
Xiaodong Liu
Oleksandr Polozov
Matthew Richardson

title: RAT-SQL: Relation-aware schema encoding and linking for text-to-SQL parsers

venue: In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 7567–7578,

authors:
Yushi Wang
Jonathan Berant
Percy Liang

title: Building a semantic parser overnight

venue: Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers),

authors:
Thomas Wolf
Lysandre Debut
Victor Sanh
Julien Chaumond
Clement Delangue
Anthony Moi
Pierric Cistac
Tim Rault
R’emi Louf
Morgan Funtowicz
Jamie Brew

title: Huggingface’s transformers: State-of-the-art natural language processing

authors:
Pengcheng Yin
Graham Neubig
Wen-tau Yih
Sebastian Riedel

title: TaBERT: Pretraining for joint understanding of textual and tabular data

title: Tabert: Pretraining for joint understanding of textual and tabular data

venue: arXiv preprint arXiv:2005.08314,

authors:
Tao Yu
Michihiro Yasunaga
Kai Yang
Rui Zhang
Dongxu Wang
Zifan Li
Dragomir Radev

title: Syntaxsqlnet: Syntax tree networks for complex and cross-domain text-to-sql task

venue: In Proceedings of EMNLP. Association for Computational Linguistics,

authors:
Tao Yu
Rui Zhang
Kai Yang
Michihiro Yasunaga
Dongxu Wang
Zifan Li
James Ma
Irene Li
Qingning Yao
Shanelle Roman
Zilin Zhang
Dragomir Radev

title: Spider: A large-scale human-labeled dataset for complex and cross-domain semantic parsing and text-to-sql task

authors:
Luke S. Zettlemoyer
Michael Collins

title: Learning to map sentences to logical form: Structured classification with probabilistic categorial grammars

authors:
Li Zhang
Shuo Zhang
Krisztian Balog

title: Table2vec: Neural word and entity embeddings for table population and retrieval

venue: In Proceedings of the 42Nd International ACM SIGIR Conference on Research and Development in Information Retrieval,

authors:
Rui Zhang
Tao Yu
He Yang Er
Sungrok Shim
Eric Xue
Xi Victoria Lin
Tianze Shi
Caiming Xiong
Richard Socher
Dragomir Radev

title: Editing-based sql query generation for cross-domain context-dependent questions

venue: In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and 9th International Joint Conference on Natural Language Processing. Association for Computational Linguistics,

authors:
Victor Zhong
Caiming Xiong
Richard Socher

title: Seq2sql: Generating structured queries from natural language using reinforcement learning

authors:
Victor Zhong
M. Lewis
Sida I. Wang
Luke Zettlemoyer

title: Grounded adaptation for zero-shot executable semantic parsing

venue: The 2020 Conference on Empirical Methods in Natural Language Processing,

heading: 1 INTRODUCTION

text:
Tabular data serve as important information source for human decision makers in many domains, such as finance, health care, retail and so on. While tabular data can be efficiently accessed via the structured query language (SQL), a natural language interface allows such data to be more accessible for a wider range of non-technical users. As a result, table semantic parsing that maps natural language queries over tabular data to formal programs has drawn significant attention in recent years.
Recent pre-trained language models (LMs) such as BERT (Devlin et al., 2019) and RoBERTa (Liu et al., 2019) achieve tremendous success on a spectrum of natural language processing tasks, including semantic parsing (Zettlemoyer & Collins, 2005; Zhong et al., 2017; Yu et al., 2018b). These advances have shifted the focus from building domain-specific semantic parsers (Zettlemoyer & Collins, 2005; Artzi & Zettlemoyer, 2013; Berant & Liang, 2014; Li & Jagadish, 2014) to cross-domain semantic parsing (Zhong et al., 2017; Yu et al., 2018b; Herzig & Berant, 2018; Dong & Lapata, 2018; Wang et al., 2020; Lin et al., 2020).
Despite such significant gains, the overall performance on complex benchmarks such SPIDER (Yu et al., 2018b) and WIKITABLEQUESTIONS benchmarks are still limited, even when integrating representations of current pre-trained language models. As such tasks requires generalization to new databases/tables and more complex programs (e.g., SQL), we hypothesize that current pretrained language models are not sufficient for such tasks. First, language models pre-trained using unstructured text data such as Wikipedia and Book Corpus are exposed to a significant domain shift when directly applied to table semantic parsing, where jointly modeling the relation between utterances and structural tables is crucial. Second, conventional pre-training objectives does not consider the underlying compositionality of data (e.g., questions and SQLs) from table semantic parsing. To close this gap, we seek to learn contextual representations jointly from structured tabular data and unstructured natural language sentences, with objectives oriented towards table semantic parsing.
∗ This work was mostly done during Tao and Bailin’s internship at Salesforce Research. Victoria is now at Facebook AI.
In this paper, we propose a novel grammar-augmented pre-training framework for table semantic parsing (GRAPPA). Inspired by previous work on data synthesis for semantic parsing (Berant & Liang, 2014; Wang et al., 2015b; Jia & Liang, 2016; Herzig & Berant, 2018; Andreas, 2020), we induce a synchronous context-free grammar (SCFG) specific to mapping natural language to SQL queries from existing text-to-SQL datasets, which covers most commonly used question-SQL patterns. As shown in Figure 1, from a text-to-SQL example we can create a question-SQL template by abstracting over mentions of schema components (tables and fields), values, and SQL operations. By executing this template on randomly selected tables we can create a large number of synthetic question-SQL pairs. We train GRAPPA on these synthetic question-SQL pairs and their corresponding tables using a novel text-schema linking objective that predicts the syntactic role of a table column in the SQL for each pair. This way we encourage the model to identify table schema components that can be grounded to logical form constituents, which is critical for most table semantic parsing tasks.
To prevent overfitting to the synthetic data, we include the masked-language modelling (MLM) loss on several large-scale, high-quality table-and-language datasets and carefully balance between preserving the original natural language representations and injecting the compositional inductive bias through our synthetic data. We pre-train GRAPPA using 475k synthetic examples and 391.5k examples from existing table-and-language datasets. Our approach dramatically reduces the training time and GPU cost. We evaluate on four popular semantic parsing benchmarks in both fully supervised and weakly supervised settings. GRAPPA consistently achieves new state-of-the-art results on all of them, significantly outperforming all previously reported results.

heading: 2 METHODOLOGY

heading: 2.1 MOTIVATION

text:
Semantic parsing data is compositional because utterances are usually related to some formal representations such as logic forms and SQL queries. Numerous prior works (Berant & Liang, 2014; Wang et al., 2015a; Jia & Liang, 2016; Iyer et al., 2017; Andreas, 2020) have demonstrated the benefits of augmenting data using context-free grammar. The augmented examples can be used to teach the model to generalize beyond the given training examples.
However, data augmentation becomes more complex and less beneficial if we want to apply it to generate data for a random domain. More and more work (Zhang et al., 2019b; Herzig et al., 2020b; Campagna et al., 2020; Zhong et al., 2020) shows utilizing augmented data doesn’t always result in a
significant performance gain in cross-domain semantic parsing end tasks. The most likely reason for this is that models tend to overfit to the canonical input distribution especially the generated utterances are very different compared with the original ones.
Moreover, instead of directly training semantic parsers on the augmented data, our paper is the first to use the synthetic examples in pre-training in order to inject a compositional inductive bias to LMs and show it actually works if the overfitting problem is carefully addressed. To address the overfitting problem, in Section 2.3, we also include a small set of table related utterances in our pre-training data. We add an MLM loss on them as a regularization factor, which requires the model to balance between real and synthetic examples during the pre-training. We note that this consistently improves the performance on all downstream semantic parsing tasks (see Section 4). Finally, our pre-training method is much more data-efficient and save much more computational power than other prior work (Section 5).

heading: 2.2 DATA SYNTHESIS WITH SYNCHRONOUS CONTEXT-FREE GRAMMAR

text:
We follow Jia & Liang (2016) to design our SCFG and apply it on a large amount of tables to populate new examples. For example, as shown in Figure 1, by replacing substitutable column mentions (“locations”), table mentions (“performance”), values (“two”), and SQL logic phrases (“at least”) with the other possible candidates in the same group, our grammar generates new synthetic text-to-SQL examples with the same underlying SQL logic template. We then pre-train BERT on the augmented examples to force it to discover substitutable fragments and learn the underlying logic template so that it is able to generalize to other similar questions. Meanwhile, BERT also benefits from pre-training on a large number of different columns, table names, and values in the generated data, which could potentially improve schema linking in semantic parsing tasks.
Grammar induction To induce a cross-domain SCFG, we study examples in SPIDER since it is a publicly available dataset that includes the largest number of examples with complex compositionalities in different domains. To further show the generality of our approach, we do not develop different SCFG for each downstream task. Given a set of (x, y) pairs in SPIDER, where x and y are the utterance and SQL query respectively. We first define a set of non-terminal symbols for table names, column names, cell values, operations, etc. For example, in Table 1, we group aggregation operations such as MAX as a non-terminal AGG. We can also replace the entities/phrases with their non-terminal types in SQL query to generate a SQL production rule β. Then, we group (x, y) pairs by similar SQL production rule β. We automatically group and count Spider training examples by program templates, and select about 90 most frequent program templates β. For each program template in the grammar, we randomly select roughly 4 corresponding natural language questions, manually replace entities/phrases with their corresponding non-terminal types to create natural language templates α, and finally align them to generate each production rule ROOT→ 〈α, β〉. The manual alignment approximately takes a few hours. About 500 SPIDER examples are studied to induce the SCFG.
Data augmentation With 〈α, β〉 pairs, we can simultaneously generate pseudo natural questions and corresponding SQL queries given a new table or database. We first sample a production rule, and replace its non-terminals with one of corresponding terminals. For example, we can map the non-terminal AGG to MAX and “maximum” for the SQL query and the natural language sentence, respectively. Also, table content is used in synthesizing our pre-training data. For example, if the sampled production rule contains a value (e.g., VALUE0), we sample a value for the selected column
from the table content and add it to the SQL and question templates. This way during pre-training, GRAPPA can access the table content and learn the linking between values and columns.
We use WIKITABLES (Bhagavatula et al., 2015), which contains 1.6 million high-quality relational Wikipedia tables. We remove tables with exactly the same column names and get about 340k tables and generate 413k question-SQL pairs given these tables. Also, we generate another 62k questionSQL pairs using tables and databases in the training sets of SPIDER and WIKISQL. In total, our final pre-training dataset includes 475k question-SQL examples.
We note that SCFG is usually crude (Andreas, 2020) especially when it is applied to augment data for different domains. In this work we don’t focus on how to develop a better SCFG that generates more natural utterances. We see this as a very interesting future work to explore. Despite the fact that the SCFG is crude, our downstream task experiments show that it could be quite effective if some pre-training strategies are applied.

heading: 2.3 TABLE RELATED UTTERANCES

text: As discussed in Section 2.1, GRAPPA is also pre-trained on human annotated questions over tables with a MLM objective. We collected seven high quality datasets for textual-tabular data understanding (Table 8 in the Appendix), all of them contain Wikipedia tables or databases and the corresponding natural language utterances written by humans. We only use tables and contexts as a pre-training resource and discard all the other human labels such as answers and SQL queries.

heading: 2.4 PRE-TRAINING GRAPPA

text:
Unlike all the previous work where augmented data is used in the end task training, we apply the framework to language model pre-training. Training semantic parsers is usually slow, and augmenting a large amount of syntactic pairs directly to the end task training data can be prohibitively slow or expensive. In our work, we formulate text-to-SQL as a multi-class classification task for each column, which can be naturally combined with the MLM objective to pre-train BERT for semantic parsing. Moreover, in this way, the learned knowledge can be easily and efficiently transferred to downstream semantic parsing tasks in the exact same way as BERT (shown in Section 4).
GRAPPA is initialized by RoBERTaLARGE (Liu et al., 2019) and further pre-trained on the synthetic data with SQL semantic loss and table-related data with MLM loss. As shown in Figure 1, we follow Hwang et al. (2019) to concatenate a user utterance and the column headers into a single flat sequence separated by the </s> token. The user utterance can be either one of the original human utterances collected from the aggregated datasets or the canonical sentences sampled from the SCFG. We add the table name at the beginning of each column if there are some complex schema inputs involving multiple tables. We employ two objective functions for language model pre-training: 1) masked-language modelling (MLM), and 2) SQL semantic prediction (SSP).
MLM objective Intuitively, we would like to have a self-attention mechanism between natural language and table headers. We conduct masking for both natural language sentence and table headers. A small part of the input sequence is first replaced with the special token <mask>. The MLM loss is then computed by the cross-entropy function on predicting the masked tokens. We follow the default hyperparameters from Devlin et al. (2019) with a 15% masking probability.
SSP objective With our synthetic natural language sentence and SQL query pairs, we can add an auxiliary task to train our column representations. The proposed task is, given a natural language sentence and table headers, to predict whether a column appears in the SQL query and what operation is triggered. We then convert all SQL sequence labels into operation classification labels for each column. For example in the Figure 1, the operation classification label of the column “locations” is SELECT AND GROUP BY HAVING. In total, there are 254 potential classes for operations in our experiments.
For a column or table indexed by i, we use the encoding of the special token </s> right before it as its representation, denoted as xi to predict its corresponding operations. On top of such representations, we apply a two-layer feed-forward network followed by a GELU activation layer (Hendrycks & Gimpel, 2016) and a normalization layer (Ba et al., 2016) to the output representations. Formally, we
compute the final vector representation of each column yi by:
h = LayerNorm(GELU(W1 · xi)) yi = LayerNorm(GELU(W2 · h))
Finally, yi is employed to compute the cross-entropy loss through a classification layer. We sum losses from all columns in each training example for back-propagation. For samples from the aggregated datasets, we only compute the MLM loss to update our model. For samples from the synthetic data we generated, we compute only SSP loss to update our model. More specifically, we mix 391k natural language utterances and 475k synthetic examples together as the final pre-training data. The examples in these two groups are randomly sampled during the pre-training, and MLM loss is computed if the selected example is a natural language question, otherwise SSP for a synthetic example.

heading: 3 EXPERIMENTS

text: We conduct experiments on four cross-domain table semantic parsing tasks, where generalizing to unseen tables/databases at test time is required. We experiment with two different settings of table semantic parsing, fully supervised and weakly supervised setting. The data statistics and examples on each task are shown in Table 2 and Table 7 in the Appendix respectively.

heading: 3.1 SUPERVISED SEMANTIC PARSING

text:
We first evaluate GRAPPA on two supervised semantic parsing tasks. In a supervised semantic parsing scenario, given a question and a table or database schema, a model is expected to generate the corresponding program.
SPIDER SPIDER Yu et al. (2018b) is a large text-to-SQL dataset. It consists of 10k complex question-query pairs where many of the SQL queries contain multiple SQL keywords. It also includes 200 databases where multiple tables are joined via foreign keys. For the baseline model, we use RAT-SQL + BERT Wang et al. (2020) which is the state-of-the-art model according to the official leaderboard. We followed the official Spider evaluation to report set match accuracy.
Fully-sup. WIKISQL WIKISQL Zhong et al. (2017) is a collection of over 80k questions and SQL query pairs over 30k Wikipedia tables. We use Guo & Gao (2019), a competitive model on WIKISQL built on SQLova Hwang et al. (2019), as our base model. We adapt the same set of hyperparameters including batch size and maximum input length as in Guo & Gao (2019). For a fair comparison, we only consider single models without execution-guided decoding and report execution accuracy.

heading: 3.2 WEAKLY-SUPERVISED SEMANTIC PARSING

text:
We also consider weakly-supervised semantic parsing tasks, which are very different from SQLguided learning in pre-training. In this setting, a question and its corresponding answer are given, but the underlying meaning representation (e.g., SQL queries) are unknown.
WIKITABLEQUESTIONS This dataset contains question-denotation pairs over single Wikipedia tables Pasupat & Liang (2015). The questions involve a variety of operations such as comparisons, superlatives, and aggregations, where some of them are hard to answered by SQL queries.
Models Dev. Test Global-GNN (Bogin et al., 2019) 52.7 47.4 EditSQL (Zhang et al., 2019b) 57.6 53.4 IRNet (Guo et al., 2019) 61.9 54.7 RYANSQL (Choi et al., 2020) 70.6 60.6 TranX (Yin et al., 2020a) 64.5 - RAT-SQL (Wang et al., 2019) 62.7 57.2 w. BERT-large 69.7 65.6 w. RoBERTa-large 69.6 - w. GRAPPA (MLM) 71.1(+1.4) - w. GRAPPA (SSP) 73.6(+3.9) 67.7(+2.1) w. GRAPPA (MLM+SSP) 73.4(+3.7) 69.6(+4.0)
Table 3: Performance on SPIDER. We run each model three times by varying random seeds, and the average scores are shown.
Models Dev. Test (Dong & Lapata, 2018) 79.0 78.5 (Shi et al., 2018) 84.0 83.7 (Hwang et al., 2019) 87.2 86.2 (He et al., 2019) 89.5 88.7 (Lyu et al., 2020) 89.1 89.2 (Guo & Gao, 2019) 90.3 89.2 w. RoBERTa-large 91.2 90.6 w. GRAPPA (MLM) 91.4 90.7 w. GRAPPA (SSP) 91.2 90.7 w. GRAPPA (MLM+SSP) 91.2 90.8 w. RoBERTa-large (10k) 79.6 79.2 w. GRAPPA (MLM+SSP) (10k) 82.3(+2.7) 82.2(+3.0)
Table 4: Performance on fully-sup. WIKISQL. All results are on execution accuracy without execution-guided decoding.
We used the model proposed by Wang et al. (2019) which is the state-of-the-art parser on this task. This model is a two-stage approach that first predicts a partial “abstract program” and then refines that program while modeling structured alignments with differential dynamic programming. The original model uses GloVe Pennington et al. (2014) as word embeddings. We modified their implementation to encode question and column names in the same way as we do in our fine-tuning method that uses RoBERTa and GRAPPA.
Weakly-sup. WIKISQL In the weakly-supervised setting of WIKISQL, only the answers (i.e., execution results of SQL queries) are available. We also employed the model proposed by Wang et al. (2019) as our baseline for this task. We made the same changes and use the same experiment settings as described in the previous section for WIKITABLEQUESTIONS.

heading: 3.3 IMPLEMENTATION OF GRAPPA

text: For fine-tuning RoBERTa, we modify the code of RoBERTa implemented by Wolf et al. (2019) and follow the hyperparameters for fine-tuning RoBERTa on RACE tasks and use batch size 24, learning rate 1e-5, and the Adam optimizer Kingma & Ba (2014). We fine-tune GRAPPA for 300k steps on eight 16GB Nvidia V100 GPUs. The pre-training procedure can be done in less than 10 hours. For all downstream experiments using GRAPPA or RoBERTa, we always use a BERT specific optimizer to fine-tune them with a learning rate of 1e-5, while using a model-specific optimizer with the respective learning rate for the rest of the base models.

heading: 4 EXPERIMENTAL RESULTS

text:
We conducted experiments to answer the following two questions: 1) Can GRAPPA provide better representations for table semantic parsing tasks? 2) What is the benefit of two pre-training objectives, namely MLM and SSP? Since GRAPPA is initialized by RoBERTa, we answer the first question by directly comparing the performance of base parser augmented with GRAPPA and RoBERTa on table semantic parsing tasks. For the second question, we report the performance of GRAPPA trained with MLM, SSP and also a variant with both of them (MLM+SSP).
Overall results We report results on the four aforementioned tasks in Tables 3, 4, 5, and 6 respectively. Overall, base models augmented with GRAPPA significantly outperforms the ones with RoBERTa by 3.7% on SPIDER, 1.8% on WIKITABLEQUESTIONS, and 2.4% on weaklysup. WIKISQL, and achieve new state-of-the-art results across all four tasks. In most cases, the combined objective of MLM+SSP helps GRAPPA achieve better performance when compared with independently using MLM and SSP. Moreover, on the low-resource setting, GRAPPA outperforms RoBERTa by 3.0% in fully-sup. WIKISQL and 3.9% in WIKITABLEQUESTIONS. Detailed results for each task are discussed as follows.
SPIDER Results on SPIDER are shown in Table 3. When augmented with GRAPPA, the model achieves significantly better performance compared with the baselines using BERT and RoBERTa.
Models Dev. Test (Liang et al., 2018) 42.3 43.1 (Dasigi et al., 2019) 42.1 43.9 (Agarwal et al., 2019) 43.2 44.1 (Herzig et al., 2020b) - 48.8 (Yin et al., 2020b) 52.2 51.8 (Wang et al., 2019) 43.7 44.5 w. RoBERTa-large 50.7(+7.0) 50.9(+6.4) w. GRAPPA (MLM) 51.5(+7.8) 51.7(+7.2) w. GRAPPA (SSP) 51.2(+7.5) 51.1(+6.6) w. GRAPPA (MLM+SSP) 51.9(+8.2) 52.7(+8.2) w. RoBERTa-large ×10% 37.3 38.1 w. GRAPPA (MLM+SSP) ×10% 40.4(+3.1) 42.0(+3.9)
Table 5: Performance on WIKITABLEQUESTIONS. Results trained on 10% of the data are shown at the bottom.
Models Dev. Test (Liang et al., 2018) 72.2 72.1 (Agarwal et al., 2019) 74.9 74.8 (Min et al., 2019) 84.4 83.9 (Herzig et al., 2020b) 85.1 83.6 (Wang et al., 2019) 79.4 79.3 w. RoBERTa-large 82.3 (+2.9) 82.3 (+3.0) w. GRAPPA (MLM) 83.3 (+3.9) 83.5 (+4.2) w. GRAPPA (SSP) 83.5(+4.1) 83.7 (+4.4) w. GRAPPA (MLM+SSP) 85.9 (+6.5) 84.7 (+5.4)
Table 6: Performance on weakly-sup. WIKISQL. We use (Wang et al., 2019) as our base model.
Our best model, GRAPPA with MLM+SSP achieves the new state-of-the-art performance, surpassing previous one (RAT-SQL+BERT-large) by a margin of 4%. Notably, most previous top systems use pre-trained contextual representations (e.g., BERT, TaBERT), indicating the importance of such representations for the cross-domain parsing task.
Fully sup. WIKISQL Results on WIKISQL are shown in Table 4. All GRAPPA models achieve nearly the same performance as RoBERTa. We suspect it is the relatively large training size and easy SQL pattern of WIKISQL make the improvement hard, comparing to SPIDER. Hence, we set up a low-resource setting where we only use 10k examples from the training data. As shown in the bottom two lines of Table 4, GRAPPA improves the performance of the SQLova model by 3.0% compared to RoBERTa, indicating that GRAPPA can make the base parser more sample-efficient.
WIKITABLEQUESTIONS Results on WIKITABLEQUESTIONS are shown in Table 5. By using RoBERTa and GRAPPA to encode question and column inputs, the performance of Wang et al. (2019) can be boosted significantly ( >6%). Compared with RoBERTa, our best model with GRAPPA (MLM+SSP) can further improve the performance by 1.8%, leading to a new state-of-the-art performance on this task. Similar to the low-resource experiments for WIKISQL, we also show the performance of the model when trained with only 10% of the training data. As shown at the bottom two lines Table 5, GRAPPA (MLM + SSP) obtains much better performance than RoBERTa, again showing its superiority of providing better representations.
Weakly sup. WIKISQL Results on weakly supervised WIKISQL are shown in Table 6. GRAPPA with MLM+SSP again achieves the best performance when compared with other baselines, obtain the new state-of-the-art results of 84.7% on this task. It is worth noting that our best model here is also better than many models trained in the fully-supervised setting in Table 4. This suggests that inductive biases injected in pre-trained representation of GRAPPA can significantly help combat the issue of spurious programs introduced by learning from denotations Pasupat & Liang (2015); Wang et al. (2019) when gold programs are not available.

heading: 5 ANALYSIS

text:
Pre-training objectives GRAPPA trained with both MLM and SSP loss consistently outperforms the one trained with one of them (MLM+SSP vs. MLM only or SSP only). GRAPPA (MLM) usually improves the performance by around 1% such as 1.4% gain on SPIDER (dev), 0.8% on WIKITABLEQUESTIONS, and 1.2% on weakly-sup. WIKISQL. By pre-training on the synthetic text-to-SQL examples, GRAPPA (SSP), we can see a similar performance gain on these tasks too except 3.9% improvement on SPIDER dev, which is what we expected (grammar is overfitted to SPIDER). By pre-training with both MLM and SSP on the combined data, GRAPPA (MLM+SSP) consistently and significantly outperforms the one pre-trained with MLM or SSP separately (e.g., about +2% on Spider, +1.5% on WikiTableQuestions, and +1.2% on weakly-sup WikiSQL.). This contributes to our key argument in the paper: in order to effectively inject compositional inductive bias to LM, pre-training on synthetic data should be regularized properly (using SSP+MLM together
instead of SSP or MLM only) in order to balance between preserving the original BERT encoding ability and injecting compositional inductive bias, otherwise, the improvements are not robust and limited (using SSP or MLM only).
Generalization As mentioned in Section 2.2, we design our SCFG solely based on SPIDER, and then sample from it to generate synthetic examples. Despite the fact that GRAPPA pre-trained on such corpus is optimized to the SPIDER data distribution, which is very different from WIKISQL and WIKITABLEQUESTIONS, GRAPPA is still able to improve performance on the two datasets. In particular, for WIKITABLEQUESTIONS where the underlying distribution of programs (not necessarily in the form of SQL) are latent, GRAPPA can still help a parser generalize better, indicating GRAPPA can be beneficial for general table understanding even though it is pre-trained on SQL specific semantics. We believe that incorporating rules from a broader range of datasets (e.g. WIKITABLEQUESTIONS) would further improve the performance. However, in this paper, we study rules from only the SPIDER dataset and test the effectiveness on other unseen datasets with different different underlying rules on purpose in order to show the generality of our method.
Even though GRAPPA is pre-trained on synthetic text-to-SQL data, the proposed pre-training method can also be applied to many other semantic parsing tasks with different formal programs (e.g., logic forms); and we also demonstrated the effectness of GRAPPA on non text-to-SQL tasks (weaklysupervised WIKISQL and WIKITABLEQUESTIONS where no programs are used, training is supervised by only answers/cell values) the underlying distribution of programs (not necessarily in the form of SQL) are latent. Furthermore, to design the SCFG and synthesize data with the corresponding programs labeled, we can use any formal programs such as the logic form or SParQL, and then employ the data to pre-train GraPPa. In this paper we choose SQL as the formal program to represent the formal representation of the questions simply because more semantic parsing datasets are labeled in SQL.
Pre-training time and data Our experiments on the SPIDER and WIKITABLEQUESTIONS tasks show that longer pre-training doesn’t improve and can even hurt the performance of the pre-trained model. This also indicates that synthetic data should be carefully used in order to balance between preserving the original BERT encoding ability and injecting compositional inductive bias. The best result on SPIDER is achieved by using GRAPPA pre-trained for only 5 epochs on our relatively small pre-training dataset. Compared to other recent pre-training methods for semantic parsing such as TaBERT (Yin et al., 2020a) and TAPAS (Herzig et al., 2020a), GRAPPA achieves the state-of-the-art performance (incorporated with strong base systems) on the four representative table semantic parsing tasks in less 10 hours on only 8 16GB Nvidia V100 GPUs (6 days on more than 100 V100 GPUs and 3 days on 32 TPUs for TaBERT and TAPAS respectively) Moreover, we encourage future work on studying how the size and quality of synthetic data would affect the end task performance. Also, GRAPPA (MLM+SSP) consistently outperforms other settings, which indicates that using MLM on the human annotated data is important.
Pre-training vs. training data augmentation Many recent work (Zhang et al., 2019b; Herzig et al., 2020b; Campagna et al., 2020; Zhong et al., 2020) in semantic parsing and dialog state tracking show that training models on a combination of the extra synthetic data and original training data does not improve or even hurt the performance. For example, (Zhong et al., 2020) synthesize data on training databases in several semantic parsing tasks including SPIDER, and find that training with this data augmentation leads to overfitting on the synthetic data and decreases the performance. In contrast, our pre-training approach could effectively utilize a large amount of synthesized data and improve downstream task performance. Also, the base parser with a GRAPPA encoder could usually converge to a higher performance in shorter time (see Section A.1).

heading: 6 RELATED WORK

text:
Textual-tabular data understanding Real-world data exist in both structured and unstructured forms. Recently the field has witnessed a surge of interest in joint textual-tabular data understanding problems, such as table semantic parsing (Zhong et al., 2017; Yu et al., 2018b), question answering (Pasupat & Liang, 2015; Chen et al., 2020), retrieval (Zhang et al., 2019a), fact-checking (Chen et al., 2019) and summarization (Parikh et al., 2020; Radev et al., 2020). While most work focus on
single tables, often obtained from the Web, some have extended modeling to more complex structures such as relational databases (Finegan-Dollak et al., 2018; Yu et al., 2018b; Wang et al., 2020). All of these tasks can benefit from better representation of the input text and different components of the table, and most importantly, an effective contextualization across the two modalities. Our work aims at obtaining high-quality cross-modal representation via pre-training to potentially benefit all downstream tasks.
Pre-training for NLP tasks GRAPPA is inspired by recent advances in pre-training for text such as (Devlin et al., 2019; Liu et al., 2019; Lewis et al., 2020b;a; Guu et al., 2020). Seminal work in this area shows that textual representation trained using conditional language modeling objectives significantly improves performance on various downstream NLP tasks. This triggered an exciting line of research work under the themes of (1) cross-modal pre-training that involves text (Lu et al., 2019; Peters et al., 2019; Yin et al., 2020a; Herzig et al., 2020a) and (2) pre-training architectures and objectives catering subsets of NLP tasks (Lewis et al., 2020b;a; Guu et al., 2020). GRAPPA extends these two directions further. The closest work to ours are TaBERT (Yin et al., 2020a) and TAPAS (Herzig et al., 2020a). Both are trained over millions of web tables and relevant but noisy textual context. In comparison, GRAPPA is pre-trained with a novel training objective, over synthetic data plus a much smaller but cleaner collection of text-table datasets.
Data augmentation for semantic parsing Our work was inspired by existing work on data augmentation for semantic parsing (Berant & Liang, 2014; Wang et al., 2015a; Jia & Liang, 2016; Iyer et al., 2017; Yu et al., 2018a). Berant & Liang (2014) employed a rule-based approach to generate canonical natural language utterances given a logical form. A paraphrasing model was then used to choose the canonical utterance that best paraphrases the input and to output the corresponding logical form. In contrast, Jia & Liang (2016) used prior knowledge in structural regularities to induce an SCFG and then directly use the grammar to generate more training data, which resulted in a significant improvement on the tasks. Unlike these works which augment a relatively small number of data and use them directly in end task training, we synthesize a large number of texts with SQL logic grounding to each table cheaply and use them for pre-training.

heading: 7 CONCLUSION AND FUTURE WORK

text:
In this paper, we proposed a novel and effective pre-training approach for table semantic parsing. We developed a context-free grammar to automatically generate a large amount of question-SQL pairs. Then, we introduced GRAPPA, which is an LM that is pre-trained on the synthetic examples with SQL semantic loss. We discovered that, in order to better leverage augmented data, it is important to add MLM loss on a small amount of table related utterances. Results on four semantic parsing tasks demonstrated that GRAPPA significantly outperforms RoBERTa.
While the pre-training method is surprisingly effective in its current form, we view these results primarily as an invitation for more future work in this direction. For example, this work relies on a hand-crafted grammar which often generates unnatural questions; Further improvements are likely to be made by applying more sophisticated data augmentation techniques. Also, it would be interesting to study the relative impact of the two objectives (MLM and SSP) by varying the respective number of pre-training examples. Furthermore, pre-training might benefit from synthesizing data from a more compositional grammar with a larger logical form coverage, and also from supervising by a more compositional semantic signals.

heading: A APPENDICES

text:
A.1 ADDITIONAL ANALYSIS
Training coverage As shown in Figure 2, on the challenging end text-to-SQL SPIDER task, RATSQL initialized with GRAPPA outperforms RAT-SQL using RoBERTa by about 14% in the early training stage. This shows that GRAPPA already captures some semantic knowledge in pre-training. Finally, GRAPPA is able to keep the competitive edge by 4%.
What if the task-specific training data is also used with the MLM or SSP objective in pretraining? Although we did not do the same experiments, we would like to point to the RAT-SQL paper (Wang et al., 2020) for some suggestions. They add a similar alignment loss (similar to SSP) on the SPIDER training data and found that it doesn’t make a statistically significant difference (in Appendix B).