We study the task of cross-database semantic parsing (XSP), where a system that maps natural language utterances to executable SQL queries is evaluated on databases unseen during training. Recently, several datasets, including Spider, were proposed to support development of XSP systems. We propose a challenging evaluation setup for cross-database semantic parsing, focusing on variation across database schemas and in-domain language use. We re-purpose eight semantic parsing datasets that have been well-studied in the setting where in-domain training data is available, and instead use them as additional evaluation data for XSP systems instead. We build a system that performs well on Spider, and find that it struggles to generalize to our re-purposed set. Our setup uncovers several generalization challenges for cross-database semantic parsing, demonstrating the need to use and develop diverse training and evaluation datasets. Semantic parsing is the task of mapping natural language utterances to formal meaning representations, and has been studied in tasks including instruction following, evaluating sentence meaning, and building interfaces to knowledge bases. In this paper, we focus on the task of mapping from natural language utterances to SQL queries executable in a database. Most prior work in mapping from natural language to SQL queries train and test the system on a single database. We refer to this setup as single-database semantic parsing ( SSP ). Well-studied datasets used in the SSP setting include GeoQuery (Zelle and Mooney, 1996) and ATIS (Hemphill et al., 1990; Dahl et al., 1994). as it is often cost-prohibitive to collect a suffi-cient number of training examples for all possible databases. Several datasets, including Spider (Yu et al., 2018), were proposed to evaluate this dimension of generalization. These datasets include examples grounded in multiple databases, distinguishing between training databases and evaluation databases. We refer to this setup as cross-database semantic parsing ( XSP ). While these datasets have been valuable in understanding and addressing some of the additional generalization challenges introduced by XSP, current evaluation of XSP systems has been limited to datasets designed for XSP. This limits the types of generalization challenges studied to those introduced by these datasets. Existing XSP evaluation data such as Spider simplifies some of these challenges, for example by including utterances that closely match their paired SQL query, as shown in last row of Figure 1. This setup misses an important opportunity for studying cross-database semantic parsing: evaluating on challenging datasets designed for single-database semantic parsing, like GeoQuery and ATIS. While the in-domain challenges of these datasets are relatively well-understood, generalization challenges introduced by studying these datasets in an XSP context have not been addressed. In this paper, we propose a more holistic analysis and evaluation setup for XSP. We propose to evaluate a semantic parsing system not only on evaluation data designed for XSP, but also on datasets that have only been studied in the SSP setting. Our repurposed evaluation set includes eight well-studied datasets like ATIS, but in a completely new setting. Instead of training on the original training data for these datasets, we train a single model on training data designed for the XSP setting, and evaluate the trained model on each evaluation dataset. These datasets were collected at different times, by different researchers, and with different motivations. This results in a wide variety of language usage, database structures, and SQL styles across datasets, further stressing a system's ability to adapt to unseen datasets. These variations pose many new generalization challenges for cross-database semantic parsing models, where in-domain examples are not available at training time. Our proposed XSP evaluation setup addresses several evaluation challenges posed by these dataset variations. With our proposed setup, we are able to analyze the potential limitations of current cross-database semantic parsing models. We uncover and attempt to address several new forms of generalization in cross-dataset semantic parsing. We develop a neural semantic parsing model is competitive all public systems on the Spider development set, and evaluate its ability to generalize to the evaluation datasets. First, we observe that the datasets originally designed for SSP become much more difficult under the XSP setting, with a notable drop in performance from both the Spider development results. Second, we experiment with several techniques that improve generalization to the eight evaluation datasets. Finally, we provide in-depth qualitative analysis on our results. Our results and analysis demonstrate a need for diverse training and evaluation datasets for XSP. Our code and experimental setup is available at https://github.com/google-research/ language/tree/master/language/xsp . We focus on the task of semantic parsing for databases. A natural language utterance u is a sequence (cid:10) u 1 , . . . , u | u | (cid:11) , where each u i is a natural language token. The task is to map u to an executable formal query y = (cid:10) y 1 , . . . , y | y | (cid:11) executable in a database D , where each y i is a SQL query token. Single-database Semantic Parsing (SSP) In SSP, all data is grounded in the same knowledge database. The training data consists of N pairs of utterances and SQL queries { x ( l ) , y ( l ) } Nl =1 grounded in database D . The evaluation data contains M unseen pairs of utterances and SQL queries { x ( l ) , y ( l ) } Ml =1 , also grounded in D . SSP has been studied using a number of datasets including ATIS (Hemphill et al., 1990; Dahl et al., 1994) and GeoQuery (Zelle and Mooney, 1996). Many prior approaches in SSP assume access to database contents at inference time. At test time, this allows the system to resolve the columns containing novel entities by performing a database look-up; for example, by labeling entity mentions in the input utterance with the columns in which they appear (Dong and Lapata, 2016; Iyer et al., 2017; Suhr et al., 2018). Cross-database Semantic Parsing (XSP) In the XSP setting, examples from the evaluation databases are not seen at training time (Yu et al., 2018, 2019b,a). Previously, the crossdomain semantic parsing task focused mostly on databases consisting of a single table (Pasupat and Liang, 2015; Iyyer et al., 2017; Zhong et al., 2017). However, the crossdatabase setting requires generalizing to unseen domains and novel database schemas. In XSP, the N training examples are { x ( l ) , y ( l ) , D ( l ) i } Nl =1 and the M evaluation examples are { x ( l ) , y ( l ) , D ( l ) j } Nl =1 , where each D is a database. Importantly, the set of training and evaluation datasets do not overlap. In addition to the generalization challenges posed by SSP, this setting adds several challenges, including generalizing to new schema structures, domain-specific phrases, and database conventions. Unlike SSP, prior work in XSP does not assume that the system has access to database contents at model inference time (Yu et al., 2018). Preprocessing steps that perform database look-ups are unavailable at inference time. Instead, the model only has access to the database schema for each evaluation example. This setting requires additional generalization, where the model must be able to map unfamiliar entities to columns in domains unseen during training. Other Related Work Semantic parsing has been widely studied for tasks including sentence understanding (Zettlemoyer and Collins, 2005, 2007; Ba-narescu et al., 2013), instruction following (Chen and Mooney, 2011; Artzi and Zettlemoyer, 2013; Long et al., 2016; Givoli and Reichart, 2019), and knowledge base querying (Popescu et al., 2004; Poon, 2013; Iyer et al., 2017). Related to the task of semantic parsing is code generation (Oda et al., 2015; Ling et al., 2016; Yin et al., 2018; Lin et al., 2018; Iyer et al., 2018). While our experiments are performed on English-langauge data, a limited amount of existing work has explored semantic parsing in languages besides English (Wong and Mooney, 2006; Min et al., 2019). Annotating SQL queries for new domains can be expensive. Several prior works present approaches to reduce this cost, for example by having crowd-workers paraphrase generated examples (Wang et al., 2015; Zhong et al., 2017), give feedback (Iyer et al., 2017), interact with a system (Artzi and Zettlemoyer, 2011; Thomason et al., 2015; Lab-utov et al., 2018), or a combination (Herzig and Berant, 2019). Research in SSP and code generation has led to innovations including constrained decoding and grammar-based decoding (Xiao et al., 2016; Yin and Neubig, 2017; Krishnamurthy et al., 2017; Lin et al., 2019). SSP has also been studied alongside additional generalization challenges, including to new compositional structures (Finegan-Dollak et al., 2018) and with additional context (Miller et al., 1996; Zettlemoyer and Collins, 2009; Suhr et al., 2018). Recent works evaluating in the XSP setting have explored methods of jointly embedding an utterance and the database schema (Shaw et al., 2019; Bogin et al., 2019a), interactive learning (Yao et al., 2019), and using intermediate output representations and new inference methods (Herzig and Berant, 2018; Guo et al., 2019; Zhang et al., 2019; Bogin et al., 2019b; Lin et al., 2019). We incorporate several such methods proposed into our proposed system. We propose to study the task of XSP by training on datasets designed for XSP, and evaluating on datasets originally designed for SSP. In our full model, we use both the Spider 1 (Yu et al., 2018) and WikiSQL (Zhong et al., 2017) datasets for training. For evaluation, in addition to the Spider development set, 2 we use eight English-language SSP datasets curated by Finegan-Dollak et al. (2018) covering a variety of domains, for example flights, geography, and movies. 3 For each dataset, we evaluate on as much data as possible, excluding test sets. Table 1 describes our evaluation datasets. Developing evaluation metrics for these repurposed evaluation sets is challenging because of the diversity of SQL styles across different databases. Yu et al. (2018)'s proposed evaluation metric compares components of the predicted and correct query, allowing for variation in the exact form of the query, for example using different table aliases. However, it does not capture all possible SQL syntax, and fails to cover some of the gold queries in our evaluation datasets. For example, it does not handle assigning an alias to the results of an intermediate SELECT statement. Moreover, it does not measure equivalence of values, meaning 1 In addition to introducing Spider, Yu et al. (2018) propose to use a number of SSP datasets, including GeoQuery, as additional training examples for systems evaluated on Spider. However, these SSP datasets were not previously used as evaluation data in the XSP setting. During training, we use only the original Spider data, and discard this additional training data used by some Spider systems. 2 WikiSQL contains much more simplified language, SQL queries, and databases than Spider. Therefore, we focus on Spider as part of our proposed XSP evaluation setup. 3 Finegan-Dollak et al. (2018) re-split these datasets to evaluate generalization to novel query structures. However, this work still operates in the SSP setting, where in-domain training examples are available. Our setup uses the original splits of the data, rather than the structure-based splits (Table 1). not execute correctly. We propose to use variation of execution accuracy as our main metric. Execution accuracy over an evaluation set is the proportion of predicted queries which, when executed against the database, result in a table equivalent to the correct query's result. If the correct query requires ordering on the final table, we require the tables be exactly the same; if it does not, we consider result tables equivalent if they contain the same set of rows. We supplement the results with additional baselines and data filtering to address the problem of over-crediting spurious predictions. We report the empty-table prior for each dataset, demonstrating how well a model could perform if predicting incorrect queries that result in empty tables. We create a filtered subset where relative performance of systems is more meaningful, including attempting to remove examples that are impossible to solve without in-domain training data. Our heuristics include removing examples with correct queries that result in empty tables, and where the correct query contains a value token that is not copiable from the input. 4 For example, in Restaurants, the phrase good restaurant always requires constraining the SQL query to restaurants with a rating greater than 2.5, even when the rating is not explicitly mentioned. Single-database semantic parsing requires recognizing unseen, in-domain entities, understanding new compositional structures, and generating executable representations. Cross-database semantic 4 Details are included in Appendix A. parsing introduces additional challenges, which we analyze and discuss below. We find that with existing XSP datasets, these challenges have been relatively under-explored. In our proposed setup, where we evaluate on datasets designed for SSP, these challenges become more prevalent. Generalizing to a new domain requires understanding domain-particular language, including entity mentions and their types, and how to map domain-specific phrases to SQL queries. Identifying Entities In the XSP setting, identifying spans of tokens comprising relevant entities in the utterance is difficult, especially without access to the database contents. For example, in some databases, first and last names are stored in separate columns, so the corresponding tokens should appear in different parts of the SQL query. In other databases, a single column is used to store names. Even if a model is trained on databases where names are always stored in a single column, it still must generalize to databases where first and last names are stored in separate columns. This becomes more challenging with domain-specific entities. For example, in the Advising example in Figure 1, the span EECS 478 refers to two distinct database entities, rather than a single entity. This requires taking into account the database schema, for example by considering that the course table has distinct columns for department and number . a natural language utterance to an executable SQL query requires correctly identifying which columns and tables each entity should be compared with. Consider the following example (from GeoQuery): NL : what states are next to the mississippi SQL : select traverse from river where river name = mississippi'; To correctly identify that the entity mississippi refers to a river name in the river table, the system must have some domain knowledge. mississippi appears twice in the database: as a state and as a river. Even in the SSP setup, if the system has access to database contents, this entity mention's type is ambiguous without reasoning about its context in the utterance. In the XSP setup, this problem becomes even more difficult. Database contents are not available at model inference time, so an exhaustive search over the database for matching columns is not possible. Without in-domain training data, the model must still be able to choose the most likely column match for each mentioned entity. However, sometimes the column name is explicitly mentioned in the utterance, making the matching problem much easier, as demonstrated in the Spider example of Figure 1. We measure how prevalent the challenge of mapping from entities to column names is in our XSP setup. In each evaluation set, we estimate the proportion of examples whose entity mentions can be resolved using exact string matching between the utterance and the schema. 5 Yavuz et al. (2018) perform a similar analysis manually on the WikiSQL dataset, estimating that roughly 54.1% of examples can be solved using exact match. The rightmost column in Table 1 compares all eight evaluation datasets and the Spider development set. In all eight evaluation datasets originally developed for SSP, fewer than half of examples explicitly mention column names for all entities in the utterance. In contrast, all column names are explicitly mentioned in least 72.4% of examples in the Spider development set. These results demonstrate that addressing this challenge is critical to XSP on completely unseen domains. Domain-Specific Phrases Generalizing to new domains requires mapping domain-specific phrases to implementations in SQL. Consider the following examples (from GeoQuery): 5 More details on this analysis are available in Appendix A. NL : what is the smallest city in arkansas SQL : select city name from city where population = (select min (population) from city where state name = arkansas') and state name = arkansas' NL : what is the smallest state that borders texas SQL : select state name from state where area = (select min (area) from state where state name in (select border from border info where state name = texas')) and state name in (select border from border info where state name = texas') When smallest describes a city, it requires sorting by the city.population column, but when used to describe a state, it requires sorting by the state.area column, even though the state table also has a population column. Another phrase whose implementation may change in a new database is how many . This phrase is often mapped to the count operator, but is sometimes mapped to specific database columns. For example, in Figure 1, how many credits maps to the credits table in Advising, and how many people maps to the population table in GeoQuery. To scope the problem, Yu et al. (2018) avoid including examples in Spider that require commonsense reasoning, including examples of domain-specific phrases. However, understanding domain-specific phrases is an important capability for a domain-general semantic parsing system. Cross-database semantic parsing requires generalizing to new database schemas, including larger tables and compositions of SQL components. Four of our evaluation datasets have at least ten tables in the database, with the largest database being ATIS with 32 tables. 6 Figure 1 demonstrates that generating queries for large databases such as ATIS often requires reasoning about the relationships between many tables. In contrast, our training databases are relatively small, with one table per example in WikiSQL and an average of 5.1 per database in Spider. The queries themselves also vary in complexity. Finegan-Dollak et al. (2018) show that our eight target evaluation datasets range from using 1.4 to 6.4 tables per SQL query. We estimate that in Spider, an average query uses around 1.7 tables, which 6 Finegan-Dollak et al. (2018) and Yu et al. (2018) provide comprehensive statistics on the databases and gold queries in our evaluation domains. is more than only one target dataset (GeoQuery). Generalizing to new databases and datasets in our setting requires generating queries that use more tables than the training data. In some evaluation datasets, the system must not only reason about the input utterance and schema, but about dataset-specific conventions that are not specified in the inputs. Consider the following example (from Scholar): NL : papers on semantic parsing SQL : select distinct T1.paperid from keyphrase as T2, paper as T1, paperkeyphrase as T3, where T2.keyphrasename = semantic parsing' and T3.keyphraseid = T2.keyphraseid and T1.paperid = T3.paperid; The annotated SQL query for this utterance returns the paperid column from the paper table. However, the paper table also includes a column named title . The utterance does not specify whether the final column should be paperid or title . While both columns may seem like reasonable options, the dataset's convention is that a list of papers should be presented using the paperid column, and a query selecting the title column will have an incorrect execution result. Such conventions are difficult, if not impossible, to learn without any in-domain training data. Unfortunately, these cases occur in nearly all target datasets. We do not focus on addressing this type of generalization, and instead report how pervasive this problem is during error analysis. A possible direction for future work is to assume access to a small number of in-domain training examples and perform few-shot learning. Our model takes as input an utterance x and a database schema S . Similar to Guo et al. (2019), we serialize S into a sequence of wordpieces s = t 0 + t 1 +    + t |S| . Each t i is a serialization of table T i , where t i = (cid:104) TAB (cid:105) + T i + c i, 0 + c i, 1 + . . . c i, |T i | . TAB is a token noting the beginning of a table schema serialization. T i is the tokenization of table T i 's name. Each c i,j is a serialization of a column C i,j , where c i,j = CT i,j + C i,j . CT i,j is a token denoting the type of the column's contents as provided by the database schema, for example numerical or text. C i,j is the tokenization of the column's name. The ordering of table schemas in s and table columns in each t i is arbitrary. 7 The input to the encoder is the concatenation of the query wordpieces and the serialized schema, represented as the sequence of tokens x = (cid:104) CLS (cid:105) + u + (cid:104) SEP (cid:105) + s . The inputs to the encoder are embedded and passed to a pretrained Transformer encoder such as BERT (De-vlin et al., 2019). The decoder is an autoregressive Transformer decoder (Vaswani et al., 2017) that attends over the outputs of the encoder and the generated prefix. We use a training set { x ( l ) , y ( l ) , S ( l ) } Nl =1 consisting of pairs of natural language utterances, gold SQL queries, and database schemas. We train the encoder and decoder end-to-end, minimizing the token-level cross-entropy loss of the gold query y ( l ) . We update the parameters of the pre-trained encoder during training. For training data we use training sets developed for XSP. Importantly, to ensure we are evaluating the cross -database setting, our training data does not include examples from the evaluation databases. During inference, we use beam search and execute the highest-probability, syntactically correct prediction. We impose a maximum execution time of 45 seconds for predictions. More details on the model, learning, and evaluation setup are available in Appendix B. 5.1 Generalization Strategies While using pre-trained language models can help encode natural language text, we need other strategies to reason jointly about the language and the database schema in completely unseen domains. We focus on generalizing to domain-specific language and novel database structures. Value Copying Similar to previous work (Jia and Liang, 2016; Gu et al., 2016; Gulcehre et al., 2016; See et al., 2017), we use a copy mechanism in the decoder. At each output step, the decoder generates a distribution over possible actions, including selecting a symbol from the output vocabulary, and copying a token from the input x . We only allow copying of certain token types, and mask out invalid copying actions, including independent wordpieces from u and TAB and column-type tokens. For table and column tokens, the name of 7 To discourage over-fitting to an arbitrary ordering of schema elements, we duplicate each Spider training example seven times with randomly permuted orderings. Duplicating seven times results in the number of Spider training examples roughly matching the number of WikiSQL training examples (Section 5.1). the corresponding table or column is recovered by post-processing the predicted sequence y . Previous approaches on Spider do not evaluate execution accuracy over the databases. Because the main metric does not require values in the predicted and gold queries to be the same, many approaches simplify the problem by using a placeholder token for all values during training. However, correctly generating values is critical for correctly executing predicted queries. To the best of our knowledge, our approach is the first to evaluate on Spider with execution accuracy and to generate SQL queries without placeholder values. Multiple Data Sources We train with training data from Spider (Yu et al., 2018) and WikiSQL (Zhong et al., 2017). Spider includes examples of complex SQL queries grounded in multi-table databases, while queries in WikiSQL are compositionally simple and grounded in single web tables. We use WikiSQL to improve generalization to domain-specific data, as it covers a large variety of domains. WikiSQL contains many more tables than Spider, and prior work estimates that roughly half of WikiSQL examples require using domain knowledge to map from entity mentions to column names (Yavuz et al., 2018). Different Output Space Guo et al. (2019) demonstrated improvements on Spider by deterministically mapping SQL to an intermediate representation, SemQL, and learning to predict outputs in this space. SemQL does not require predicting all of the tables in the FROM clause of the SQL query, or explicitly predicting the columns on which tables are joined. Instead of reasoning about foreign keys, the model predicts queries in the SemQL space, which are deterministically transformed to a final SQL query. In most cases, SemQL queries can be mapped back to SQL using database foreign key relations. We implement this aspect of SemQL as a mapping from SQL to a representation with an under-specified FROM clause, which we call SQLUF . Conversion from SQL to SQLUF removes tables from the FROM clause(s) of the SQL query implicitly referenced via a column elsewhere in the query, and removes JOIN clauses. Conversion from SQLUF to SQL restores these tables, and joins between tables are inferred by greedily identifying a path that connects all tables in the FROM clause, given foreign key relations. 8 Examples of SQLUF are shown in Appendix C. 6 Experiments Comparison to Existing XSP Systems Our best model performs well on the Spider development set. Table 3 compares our system with top systems on the Spider leaderboard 9 . On the development set, our model performs competitively with contemporaneous systems. Table 3 shows that Spider performance correlated to the choice of the pre-trained models. Public BERTLARGE is better than BERTBASE . To further improve performance, we experiment with an enhanced pre-trained model BERTLARGE + following the recipe proposed by Liu et al. (2019). The BERTLARGE + model is trained with 8K batch size and 100k training steps, and in contrast to RoBERTa, is only trained on the Wiki + Books Corpus used in Devlin et al. (2019). Training our model to predict value placeholders ( Value Copying) instead of copying values from the input results in a performance drop, showing a benefit of modeling values even when ignored by the metric. 10 XSP on Unseen Datasets Table 2 shows results on all evaluation data, including datasets originally studied in the SSP setting. We report results on the filtered set (Section 3) as well as the full set of these datasets. A large portion of the examples in datasets such as Restaurants and Advising yield empty execution results. This shows the need to also evaluate on the filtered set, where incorrect spurious predictions are much less likely to result in the same table as a gold query with an empty table result. Second, while execution accuracy on Spider is relatively high, performance on the other evaluation datasets is much lower. We find that all three techniques for addressing generalization challenges are effective. First, including WikiSQL in the training data results in better performance than only using Spider training data. We hypothesize that this is due to the addi-8 Like SemQL, this conversion is not possible if foreign key relations between predicted tables are not provided or if a given table is referenced more than once in a FROM clause. This can also result in a lossy or ambiguous conversion if there are multiple foreign key relations between a pair of tables. 9 https://yale-lily.github.io/spider . In Table 3, we include non-anonymized leaderboard submissions, and for anonymous systems, the most recent submission for duplicate systems. 10 About 55% of examples in Spider do not require copying values from the input utterance to the gold query. tional domains in WikiSQL, as well as the larger proportion of examples that require mapping from entities in the utterance to column names (Yavuz et al., 2018). Using SQLUF also improves performance, as it produces queries coherent with respect to the schema, for example only selecting columns from tables where the column exists. Finally, using value placeholders significantly reduces execution accuracy in all datasets. While masking values decreases Exact Set Match on Spider by 10.9%, its effect on execution accuracy can be devastating both for Spider and the eight evaluation datasets. This demonstrates the need to consider execution results when evaluating semantic parsing systems. Error Analysis For each evaluation dataset, we analyze twenty random predictions from the filtered subset. Examples of the most common error types are shown in Figure 2, along with the proportion of analyzed predictions in the eight target datasets that contain the error type. Appendix D discusses the complete results of error analysis. 40% of errors are caused by comparing an entity to the wrong column, for example searching for James Bond' in the director.name column when it actually refers to a movie.title . This usually requires using domain knowledge identify to columns that are likely to contain the mentioned entity (Section 4.1). 31.1% of errors are caused by missing constraints specified in the utterance, for example by failing to use a relevant entity in the predicted query. 28.8% of errors are also caused by incorrectly identifying entity spans, for example by treating FIN 340 as a single entity rather than two separate entities in the database (Section 4.1). Another common error is predicting the wrong final column. While choosing what to return is difficult for the model due to understanding domain-specific phrases such as how many (20.0% of errors; Section 4.1), sometimes the errors are due to dataset conventions (26.9% of errors; Section 4.3). For example, the paperid column should be selected instead of the title column in Scholar. Such dataset conventions could be learned through few-shot learning, where a small number of in-domain training examples are available. Our system is required to generalize to larger databases than it was trained on, including more complex compositions of tables (Section 4.2). For 40.0%  Entity-column matching (IMDBXSP ) NL : List James Bond directors Pred. example, while SQLUF can be used to represent most gold queries in most evaluation datasets (shown in Appendix C), in ATIS, only 17.3% of gold queries are covered by SQLUF . Most of the uncovered examples require mapping two columns, to airport and from airport , in the same table flight to the same foreign key airport service.airport code . This compositional structure is not covered by SQLUF , but is critical to perform well on ATIS. We study the task of cross-database semantic parsing (XSP), where a system that maps natural language utterances to executable SQL queries is evaluated on databases unseen at training time. While this task has been studied through datasets developed specifically for XSP, we propose a more holistic evaluation for XSP, where we also evaluate on datasets originally studied in a setting where in-domain training data is available. We identify several new generalization challenges that arise when evaluating in our proposed setup, including identifying entities, mapping entities and domain-specific phrases to a database schema, and generalizing to more complex database schemas. Using a model that performs well on evaluation data designed for XSP, we are able to move towards addressing some of the generalization challenges on these additional evaluation sets without any in-domain training data. Our results and analysis demonstrate the need for developing more holistic evaluation of cross-database semantic parsing using a more diverse set of language and databases. Several significant generalization challenges remaining, including improving commonsense and in-domain reasoning and table schema understanding capabilities. Some examples in our filtered evaluation set still require reasoning about dataset conventions that are difficult to acquire without in-domain training examples. Future work could also make the stronger assumption that a small number of in-domain training examples are available, and train and evaluate in a few-shot setting. The first author is supported by the National Science Foundation Graduate Research Fellowship under Grant No. DGE-1650441. We thank the Google Research Language and Cornell NLP groups for their comments and feedback during the project's development. We also thank Jing Li for pre-training the BERTLARGE + model, and Philip Massey, Zuyao Li, Angelica Chen, Karl Pichotta, and Francesco Piccinno for their contributions to the codebase. Finally, we thank the anonymous reviewers for their comments and suggestions.
method abstain objective objective result objective abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain objective objective objective method abstain abstain abstain objective objective objective objective objective result result objective other method other other other other other other other other other other other other other other other other other other other other abstain other other other other other objective abstain method method method abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method method abstain abstain abstain abstain abstain abstain other abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method method method abstain method method abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain method method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain other abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain method abstain abstain method abstain abstain method abstain other other other other other