The robustness of Text-to-SQL parsers against adversarial perturbations plays a crucial role in delivering highly reliable applications. Previous studies along this line primarily focused on perturbations in the natural language question side, neglecting the variability of tables. Motivated by this, we propose the A dversarial T able P erturbation ( ATP ) as a new attacking paradigm to measure the robustness of Text-to-SQL models. Following this proposition, we curate ADVETA, the first robustness evaluation benchmark featuring natural and realistic ATPs. All tested state-of-the-art models experience dramatic performance drops on ADVETA, revealing models' vulnerability in real-world practices. To defend against ATP, we build a systematic adversarial training example generation framework tailored for better contextualization of tabular data. Experiments show that our approach not only brings the best robustness improvement against table-side perturbations but also substantially empowers models against NL-side perturbations. We release our benchmark and code at: https://github.com/microsoft/ContextualSP. The goal of Text-to-SQL is to generate an executable SQL query given a natural language (NL) question and corresponding tables as inputs. By helping non-experts interact with ever-growing databases, this task has many potential applications in real life, thereby receiving considerable interest from both industry and academia (Li and Jagadish, 2016; Zhong et al., 2017; Affolter et al., 2019). Recently, existing Text-to-SQL parsers have been found vulnerable to perturbations in NL questions (Gan et al., 2021; Zeng et al., 2020; Deng et al., 2021). For example, Deng et al. (2021) removed the explicit mentions of database items in a  Equal contributions during the internship at Microsoft Research Asia. Student Name Citizenship Score Semester A Country X 92 Fall B Country Y 90 Spring A Country X 89 B Country Y 85 Fall C Country Z 97 Spring Original Table List names and citizenships of students who achieved top 3 scores . SELECT Student Name, Citizenship FROM Student ORDER BY Score desc LIMIT 3 SELECT Student Name FROM Student ORDER BY Score desc LIMIT 3 (Missing Nationality) SELECT Student Name, Instructor Name, Citizenship FROM Student ORDER BY Grade desc LIMIT 3 Student Name Nationality Score Semester A Country X 92 Fall B Country Y 90 Spring A Country X 89 Spring B Country Y 85 Fall C Country Z 97 Spring RPLP e r t u r b e d T ab l e Student Name Citizenship Score Semester Instructor Name Grade A Country X 92 Fall D 6 B Country Y 90 Spring E 6 A Country X 89 Spring E 6 B Country Y 85 Fall D 5 C Country Z 97 Spring F 5 ADDP e r t u r b e d T ab l e Figure 1: Adversarial examples based on table perturbations for a Text-to-SQL parser. question while keeping its meaning unchanged, and observed a significant performance drop of a Text-to-SQL parser. Gan et al. (2021) also observed a dramatic performance drop when the schema-related tokens in questions are replaced with synonyms. They investigated both multi-annotations for schema items and adversarial training to improve parsers' robustness against permutations in NL questions. However, previous works only studied the robustness of parsers from the perspective of NL questions, neglecting variability from the other side of parser input  tables. We argue that a reliable parser should also be robust against table-side perturbations since they are inevitably modified in the human-machine interaction process. In business scenarios, table main-2007 tainers may (i) rename columns due to business demands and user preferences. (ii) add new columns into existing tables when business demands change. Consequently, the extra lexicon diversity introduced by such modifications could harm performances of unrobust Text-to-SQL parsers. To formalize these scenarios, we propose a new attacking paradigm, A dversarial T able P erturbation ( ATP ), to measure parsers' robustness against natural and realistic ATPs. In accordance with the two scenarios above, we consider both R E PL ACE ( RPL ) and ADD perturbations in this work. Figure 1 conveys an intuitive understanding of ATP. Ideally, ATP should be conducted based on two criteria: (i) Human experts consistently write correct SQL queries before and after table perturbations, yet parsers fail; (ii) Perturbed tables look natural and grammatical, and are free from breakage of human language conventions. Accordingly, we carefully design principles for RPL/ADD and manually curate the ADVE rsarial T able perturb A tion ( ADVETA ) benchmark based on three existing datasets. All evaluated state-of-the-art Text-to-SQL models experience drastic performance drops on ADVETA: On ADVETA-RPL, the average relative percentage drop is as high as 53 . 1% , whereas on ADVETA-ADD is 25 . 6% , revealing models' lack of robustness against ATPs. Empirically, model robustness can be improved by adversarial training, i.e. re-train models with training set augmented with adversarial examples (Jin et al., 2020). However, due to the different natures of structured tables and unstructured text, well-established text adversarial example generation approaches are not readily applicable. Motivated by this, we propose an effective C ontextualized T able A ugmentation (CTA) approach that better leverages tabular context information and carry out ablation analysis. To summarize, the contributions of our work are three-fold:  To the best of our knowledge, we are the first to propose definitions and principles of A dversarial T able P erturbation ( ATP ) as a new attacking paradigm for Text-to-SQL. We contribute ADVETA , the first benchmark to evaluate the robustness of Text-to-SQL models. Significant performance drops of state-of-the-art models reveals that there is much more to explore beyond high leader-board scores. We design CTA , a systematic adversarial training example generation framework tailored for better contextualization of tabular data. Experiments show that our approach brings model best robustness gain and lowest original performance loss , compared with various baselines. Moreover, we show that adversarial robustness brought by CTA generalizes well to NL-side perturbations. We propose the A dversarial T able P erturbation ( ATP ) paradigm to measure robustness of Text-to-SQL models. For an input table and its associated NL questions, the goal of ATP is to fool Text-to-SQL parsers by perturbing tables naturally and realistically. More specifically, human SQL experts can consistently maintain their correct translations from NL questions to SQL with their understanding of language and table context. Formally, ATP consists of two approaches: R E PL ACE ( RPL ) and ADD . In the rest of this section, we first discuss our consideration of table context, then introduce conduction principles of RPL and ADD. Tables consist of explicit and implicit elements  both are necessary for understanding table context. Explicit elements refer to table captions, columns, and cell values. Implicit elements, in our consideration, contains T able P rimary E ntity ( TPE ) and domain. (Relational) Tables are structured data recording domain-specific attributes (columns) around some central entities (TPE) (Sumathi and Esakkirajan, 2007). Without the explicit annotation, humans could still make correct guesses on them. For example, it's intuitive that tables in Figure 1 can be classified as education domain, and all of the columns center around the TPE student. Combining both explicit and implicit elements, people achieve an understanding of table context, which becomes the source of lexicon diversity in column descriptions. Given a target column, the goal of RPL is to seek an alternative column name that makes sense to humans but misleads unrobust models. Gold SQL, as illustrated in Figure 1, should be correspondingly adapted by mapping the original column to its new name. In light of this, RPL should fulfill the following two principles: 2008 ADVETA Statistics Spider WTQ WikiSQL Orig. Semantic Equivalency: Under the table context of target column, substituted column names are expected to convey equivalent semantic meaning as the original name. Phraseology Correctness: ATP aims to be natural and realistic and does not target worst-case attacks. Therefore, replaced column names are expected to follow linguistic phraseology conventions: (i) Grammar Correctness: Substituted column names should be free from grammar errors. (ii) Proper Collocation with TPE: New column names should collocate properly with TPE. For example, height and tallness both collocate well with student (TPE), but conventionally not altitude . (iii) Idiomaticity: New column names should sound natural to a native speaker to address target columns. For example, runner-up means second place , and racer-up is a bad replacement despite runner is synonymous to racer . ADD perturbs tables with introductions of new columns. Instead of adding random columns that fit well into the table domain, we pertinently add adversarial columns with respect to a target column for the sake of adversarial efficiency. Gold SQL should remain unchanged after ADD perturbations 1 . Below states ADD principles: Semantic-association & Domain-relevancy: Given a target column and its table context, newly added columns are expected to (i) fit nicely into the table context; (ii) have high semantic associations with the target column yet low semantic equivalency (e.g. sales vs. profits , editor vs. author ). Phraseology Correctness: Same as RPL, columns should obey human language conventions. Irreplaceability: Unlike RPL, any added 1 We omit cell value alignment in ADD for simplicity. columns should be irreplaceable with any original table columns. In other words, ADD requires semantic equivalency to be filtered out from highly semantic associations. Otherwise, the original gold SQL will not be the single correct output, which makes the perturbation unreasonable. Following RPL and ADD principles, we manually curate the ADVE rsarial T able perturb A tion ( ADVETA ) benchmark based on three mainstream Text-to-SQL datasets, Spider (Yu et al., 2018), WikiSQL (Zhong et al., 2017) and WTQ (Paper-not et al., 2017). For each table from the original development set , we conduct RPL/ADD annotation separately, perturbing only table columns. For its associated NL-SQL pairs, we leave the NL questions unchanged and adapt gold SQLs accordingly. As a result, ADVETA consists of 3 (Spi-der/WTQ/WikiSQL)  2 (RPL/ADD) = 6 subsets. We next introduce annotation details and characteristics of ADVETA. Five vendors join the annotation process. Each base dev set is split into small chunks and is manually annotated by one vendor and reviewed by another, with an inter-annotator agreement to resolve annotation inconsistency. Before annotation, vendors are first trained to understand table context as described in  2, then are further instructed of the following details. RPL : RPL principles are the mandatory requirements. During annotation, vendors are given full Google access to ease the conception of synonymous names for a target column. ADD : ADD principles will be the primary guideline. Unlike freestyle RPL annotations, vendors are provided with 2009 StudentName Citizenship Score Age A Country X 92 19 B Country Y 89 21 StudentName Citizenship Score School Term A Country X 92 Fall B Country Y 89 Spring Student Name Citizenship Score Academic Year A Country X 92 Fall B Country Y 89 Spring Candidate Tables WDC Dense Retrieval TAPAS Reranker Number-batch StudentName Citizenship Score Semester A Country X 92 Fall B Country Y 89 Spring Caption: School Scores Statistics Top K Similar Tables ... ... ... ... a list of 20 candidate columns from where they select 3-5 based on semantic-association 2 Notice that we only consider columns mentioned at least once across NL questions to avoid vain efforts. In Appendix A, We display some representative benchmark annotation cases. We present comprehensive benchmark statistics and analysis results in Table 1. Notice that we limit the scope of statistics only to perturbed columns (as marked by #Avg. perturbed col per table ). Basic Statistics reflects elementary information of ADVETA. Analytical Statistics illustrate highlighted features of ADVETA compared with original dev-sets: (i) Diverse column names for a single semantic meaning: each table from the RPL subset contains approximately five times more lexicons which are used to express a single semantic meaning 3 . (ii) Table concept richness: each table from ADD subset contains roughly five times more columns with unique semantic meanings. In this section, we introduce our C ontextualized T able A ugmentation (CTA) framework as an adversarial training example generation approach tailored for tabular data. The philosophy of adversarial example generation is straightforward: Pushing 2 We generate the candidate list with a retriever-reranker combo from 4. 3 For example, column names { Last name, Family name, Surname } express a single semantic meaning. In practice, we random sample at most 100 tables from each split, and obtain the number of unique semantic meanings by manual count. augmented RPL/ADD lexicon distributions closer to human-agreeable RPL/ADD distributions. This requires maximization of lexicon diversity under the constraints of domain relevancy and clear differentiation between semantic association & semantic equivalency, as stated in ADD principle from  2. Well-established text adversarial example generation approaches, such as TextFooler (Jin et al., 2020) and BertAttack (Li et al., 2020), might fail to meet this objective because: (i) They rely on syntactic information (e.g. POS-tag, dependency, semantic role) to perform text transformations. However, such information is not available in structured tabular data, leading to poor-quality adversarial examples generated by these approaches. (ii) They perform sequential word-by-word transformations, which could narrow lexicon diversity (e.g. written by will not be replaced by author ). (iii) They cannot leverage tabular context to ensure domain relevancy. (iv) They generally fail to distinguish semantic equivalency from high semantic association according to our observations (e.g., fail to distinguish sales vs. profits ). To tackle these challenges, we construct the CTA framework. Given a target column from a table with NL questions, (i) a dense table retriever properly contextualizes the input table, thereby pinpointing top-k most domain-related tables (and columns) from a large-scale database while boosting lexicon diversity . (ii) A reranker further narrows down semantic-association and produces coarse-grained ADD/RPL candidates. (iii) NLI decision maker distinguishes semantic equivalency from semantic association and allocates candidate 2010 columns to RPL/ADD buckets. A detailed illustration of our CTA framework is shown in Figure 2. We next introduce each component of CTA. The entire framework starts with a dense retrieval module to gather most domain-related tables of user queries. We utilize the Tapas-based (Herzig et al., 2020) dense retriever in this module (Herzig et al., 2021), due to its better tabular contextualization expressiveness over classical retrieval methods such as Word2Vec (Mikolov et al., 2013) and BM25 (Robertson, 2009). Following the original usage proposed by Herzig et al. (2020), we retrieve the top 100 most domain-related tables from the backend Web Data Commons (WDC) (Lehm-berg et al., 2016) database consisting of 600k nonrepetitive tables with at most five columns. From these retrieved domain-related tables, we further narrow down the range of most semantically associated candidate columns. This is done by a ConceptNet Numberbatch word embedding (Speer et al., 2017) reranker, who computes the cosine similarity score for a given column pair. We choose ConceptNet Numberbatch due to its advantage of far richer (520k) in-vocabulary multi-grams compared with Word2Vec (Mikolov et al., 2013), GloVe (Pennington et al., 2014), and Counter-fitting (Mrkic et al., 2016), which is especially desirable for multi-gram columns. We keep the top 20 similar among them as RPL/ADD candidates for each column of the original table. Aside from candidates obtained from retriever-reranker for whole-column level RPL, we consider word-level RPL for a target column as a complement. Specifically, we replace each word in a given target column with its synonyms recorded in the Oxford Dictionary (noise is more controllable compared with synonyms gathered by embedding). With a probability 25% for each original word to remain unchanged, we sample until the max pre-defined number (20) of candidates is reached or 5 consecutively repeated candidates are produced. So far we have pinpointed candidate columns whose domain relevancy and semantic association are already guaranteed. The final stage is to determine which one of RPL/ADD candidates is more suitable for based on its semantic equivalent against target column. Therefore, we leverage RoBERTa-MNLI (Liu et al., 2019; Williams et al., 2017), the expert in differentiating semantic equivalency from semantic association 4 . Practically, we construct premise-hypothesis by contextualized columns and judge semantic equivalency based on output bidirectional entailment scores e 1 and e 2 . NLI Premise-Hypothesis Construction The Quality of premise-hypothesis plays a key factor for NLI's functioning. We identify three potentially useful elements for contextualizing columns with surrounding table context: TPE, column type, and column cell value. Through manual experiments, we observe that: (i) Adding cell value significantly hurt decision accuracy of NLI models. (ii) TPE is the most important context information and cannot be ablated. (iii) Column type information can be a desirable source for word-sense disambiguation. Thus the final template for premise-hypothesis construction as python formatted string is expressed as: f  { TPE } { CN } ( { CT } ) .  , where CN is column name, and CT is column type. RPL/ADD Decision Criterion In practice, we observe a discrepancy in output entailment scores between premise-hypothesis score e 1 and hypothesis-premise score e 2 . Thus we take scores from both direction into consideration. For RPL, we empirically choose min ( e 1 , e 2 ) > = 0 . 65 (Figure 2) as the final RPL acceptance criterion to reduce occurrences of false positive entailment decision. For ADD, the criterion is instead max ( e 1 , e 2 ) < = 0 . 45 to reduce false negative entailment decisions 5 . Datasets and Models The five original Text-to-SQL datasets involves in our experiments are: Spider (Yu et al., 2018), WikiSQL (Zhong et al., 2017), WTQ (Shi et al., 2020) 6 , CoSQL (Yu et al., 2019a) and SParC (Yu et al., 2019b). Their corresponding perturbed tables are from our ADVETA 4 We highly recommend reading our pilot study in B.1. 5 To avoid semantic conflict between a new column  c and original columns c 1 ,    , c n , we apply to each pair of ( c, c i ) . 6 Note that we use the version with SQL annotations provided by Shi et al. (2020) here, since the original WTQ (Pasupat and Liang, 2015) only contains answer annotations. benchmark. WikiSQL and WTQ are single-table, while Spider, CoSQL, and SParC have multi-tables. CoSQL and SParC are known as multi-turn Text-to-SQL datasets, sharing the same tables with Spider. Dataset statistics are shown in Appendix Table 11. We evaluate open-source Text-to-SQL models that reach competitive performance on the aforementioned datasets. DuoRAT (Scholak et al., 2021) and ETA (Liu et al., 2021) are baselines for Spider; SQUALL (Shi et al., 2020) is the baseline for WTQ; SQLova (Hwang et al., 2019) and CESQL (Guo and Gao, 2019) are baselines for WikiSQL. For the two multi-turn datasets (CoSQL & SParC), baselines are EditSQL (Zhang et al., 2019) and IGSQL (Cai and Wan, 2020). Exact Match ( EM ) is employed for evaluation metric across all settings. Training details are shown in C.2. Attack Details All baseline models are trained from scratch on corresponding original training sets, and then independently evaluated on original dev sets, ADVETA-RPL and ADVETA-ADD. Since columns have around three manual candidates in ADVETA-RPL/ADD, the number of possible perturbed tables scales exponentially with column numbers for a given table from the original dev set. Therefore, models are evaluated on ADVETA-RPL/ADD by sampling perturbed tables. For each NL-SQL pair and associated table(s), we sample one RPL-perturbed table and one ADD-perturbed table in each attack. Each column mentioned from gold SQL is perturbed by a randomly sampled manual candidate from ADVETA. For performance stability and statistical significance, we run five attacks with random seeds for each NL-SQL pair. Attack Results Table 2 presents the performance of models on original dev sets, ADVETA-RPL and ADVETA-ADD. Across various task formats, domains, and model designs, state-of-the-art Text-to-SQL parsers experience dramatic performance drop on our benchmark: by RPL perturbations, the relative percentage drop is as high as 53.1%, whereas on ADD the drop is 25.6% on average 7 . Another interesting observation is that RPL consistently leads to higher performance drops than ADD. This is perhaps due to models' heavy reliance on lexical matching, instead of true understanding of language and table context. Conclusively, Text-to-SQL models are still far less robust than desired against variability from the table input side. Attack Analysis To understand the reasons for parsers' vulnerability, we specifically analyze their schema linking modules which are responsible for recognizing table elements mentioned in NL questions. This module is considered a vital component for Text-to-SQL (Wang et al., 2020; Scholak et al., 2021; Liu et al., 2021). We leverage the oracle schema linking annotations on Spider (Lei et al., 2020) and test ETA model on ADVETA using the oracle linkings. Note that we update the oracle linkings accordingly when testing on RPL. Table 4 compares the performance of ETA with or without the oracle linkings, from which we make two observations: (i) When guided with the oracle linkings, ETA performs much better on both RPL ( 27 . 6%  55 . 7% ) and ADD ( 39 . 9%  71 . 3% ). Therefore, the failure in schema linking is one of the essential causes for the vulnerability of Text-to-SQL parsers. (ii) Even with the oracle linkings, the performance of ETA on RPL and ADD still lags behind its performance on the original dev set, especially on RPL. Through a careful analysis on failure cases, we find that ETA still generates table elements that have a high degree of lexical matching with NL questions, even though the correct table elements are specified in the oracle linkings. Defense Details We carry defense experiments with SQLova, SQUALL and ETA on WikiSQL, WTQ and Spider, respectively. We compare CTA 7 Average relative performance presented in Appendix C.3. with three baseline adversarial training approaches: Word2Vec (W2V), TextFooler (TF) (Jin et al., 2020), and BERT-Attack (BA) (Li et al., 2020) (details found in D.). Models are trained from scratch on corresponding augmented training sets. Specifically, for each NL-SQL pair, we keep the original table while generating one RPL and one ADD adversarial example. As a result, augmented training data is three times as large in the sense that each NL-SQL pair is now trained against three tables: original, RPL-perturbed, and ADD-perturbed. In addition to the adversarial training defense paradigm, we also include the manual version of Multi-Annotation Selection (MAS) by Gan et al. (2021) on Spider, using their released data. The rest evaluation process is same as attack. Defense Results Table 3 presents model performance through various defense approaches. We get two observations: (i) CTA consistently brings better robustness. Compared with other approaches, CTA-augmented models have the best performance across all ADVETA-RPL/ADD settings, as well as on all original dev sets. These results demonstrate CTA can effectively improve the robustness of models against RPL and ADD perturbations while introducing fewer noises into original training sets. Interestingly, we observe that textual adversarial example generation approaches (BA, TF) are outperformed by the simple W2V approach. This verifies our analysis stated in  4. We include a case study in Appendix B.3 on characteristics of various baselines. (ii) CTA fails to bring models back to their original dev performance. Even if trained with high-quality data augmented by CTA, models could still be far worse than their original performance. This gap is highly subjected to the similarity of lexicon distribution between train and dev set. Concretely, on WikiSQL and WTQ where train and dev set have a similar domain, both RPL performance and ADD performance are brought back closer to original dev performance when augmented with CTA. On the contrary, on Spider where train-dev domains overlap less, there is still a notable gap between performance after adversarial training and the original dev performance. In conclusion, more effective defense paradigms are yet to be investigated. Defense Analysis Following attack analysis, we conduct schema linking analysis with ETA model augmented with top 2 approaches (i.e. W2V & CTA) on Spider. We follow metric calculation of (Liu et al., 2021) and details are shown in  C.4. As shown in Table 5, both approaches improve the schema linking F 1 . Specifically, CTA improves column F 1 by 3%  8% , and table F 1 by 13%  20% , compared with vanilla ETA. This reveals that improvement of robustness can be primarily attributed to better schema linking. curred by the annotation design that vendors are given CTA-retrieved candidate list for ADD annotations. However, we emphasize that: (i) RPL have NO vulnerability to data leakage since it is entirely independent of CTA. (ii) The leakage risk in ADD is negligible. On the one hand, our vast-size (600k tables) backend DB supplies tremendous data diversity, maximally reducing multiple retrievals of a single table; On the other hand, CTA's superior performance on Spider, the representative feature of which is cross-domain & cross-database across train-test splits (thus makes performance gain from data leakage hardly possible), further testifies its authentic effectiveness. We carry out an ablation study to understand the roles of two core components of CTA: dense retriever and RoBERTa-MNLI. Results are shown in Table 3. CTA w/o Retriever RPL candidates are generated merely from the dictionary; ADD generation is the same as W2V baseline. Compared with complete CTA, models augmented with this setting experience 1 . 1%  1 . 2% and 1 . 8%  7 . 6% performance drop on RPL and ADD, respectively. We attribute RPL drops to loss of real-world lexicon diversity and ADD drops to loss of domain relevancy. CTA w/o MNLI RPL and ADD candidates are generated in the same way as CTA, but without denoising of MNLI. RPL/ADD decisions solely rely on ranked semantic similarity. Compared with complete CTA, models augmented by this setting experience significant performance drops ( 4 . 9%  7 . 9% ) on all RPL subsets, and moderate drops ( 1 . 5%  2 . 8% ) on all ADD subsets. We attribute these drops to the inaccurate differentiation between semantic equivalency and semantic association due to lack of MNLI, which results in noisy RPL/ADD adversarial examples. Beyond CTA's effectiveness against table-side perturbations, a natural question follows: could retraining with adversarial table examples improve model robustness against perturbations from the other side of Text-to-SQL input (i.e., NL ques-tions)? To explore this, we directly evaluate ETA (trained with CTA-augmented Spider train-set) on Spider-Syn dataset (Gan et al., 2021), which replaces schema-related tokens in NL question with its synonym. We observe an encouraging 9 . 8% EM improvement compared with vanilla ETA (trained with Spider train-set). This verifies CTA's generalizability to NL-side perturbations , with comparable effectiveness as the previous SOTA defense approach MAS, which fails to generalize to table-side perturbations on ADVETA in Table 3. 6 Related Work Robustness of Text-to-SQL As discussed in  1, previous works (Gan et al., 2021; Zeng et al., 2020; Deng et al., 2021) exclusively study robustness of Text-to-SQL parsers against perturbations in NL question inputs. Our work instead focuses on variability from the table input side and reveals parsers' vulnerability to table perturbations. Adversarial Example Generation Existing works on adversarial text example generations can be classified into three categories: (1) Sentence-Level. This line of work generates adversarial examples by introducing distracting sentences or paraphrasing sentences (Jia and Liang, 2017; Iyyer et al., 2018). (2) Word-Level. This dimension of work generates adversarial examples by flipping words in a sentence, replacing words with their synonyms, and deleting random words (Li et al., 2020; Ren et al., 2019; Jin et al., 2020). (3) Char-Level. This line of work flips, deletes, and inserts random chars in a word to generate adversarial examples (Belinkov and Bisk, 2018; Gao et al., 2018). All the three categories of approaches have been widely used to reveal vulnerabilities of high-performance neural models on various tasks, including text classification (Ren et al., 2019; Morris et al., 2020), natural language inference (Li et al., 2020) and question answering (Ribeiro et al., 2018). Previous work on robustness of Text-to-SQL and semantic parsing models primarily adopt word-level perturbations to generate adversarial examples (Huang et al., 2014 2021). For example, the Spider-Sync adversarial benchmark (Gan et al., 2021) is curated by replacing schema-related words in questions with their synonyms. Despite these methods' effectiveness in generating adversarial text examples, they are not readily applicable for structural tabular data, as we discussed in  4. Apart from this, table-side perturbations enjoy much higher attacking efficiency : the attack coverage of a single table modification includes all affiliated SQLs, whereas one NL-side perturbation only affects a single SQL. Combined with the lighter cognitive efforts of tabular context understanding than NL-understanding, ATP is arguably lower in annotation costs . Previous work on table perturbations (Cartella et al., 2021; Ballet et al., 2019) focuses on table cell values; another work, (Ma and Wang, 2020) study impacts of naively (i.e., without consideration of table context information and without human guarantee) renaming irrelevant columns and adding irrelevant columns. In this work, we focus on table columns and propose an effective CTA framework that better leverages tabular context information for adversarial example generation, as well as manually annotate ADVETA benchmark. We introduce A dversarial T able P erturbation ( ATP ), a new paradigm for evaluating model robustness on Text-to-SQL and define its conduction principles. We curate the ADVETA benchmark, on which all state-of-the-art models experience dramatic performance drop. For defense purposes, we design the CTA framework tailored for tabular adversarial training example generation. While CTA outperforms all baseline methods in robustness enhancement, there is still an unfilled gap from the original performance. This calls for future exploration of the robustness of Text-to-SQL parsers against ATP. Our ADVETA benchmark presented in this work is a free and open resource for the community to study the robustness of Text-to-SQL models. We collected tables from three mainstream Text-to-SQL datasets, Spider (Yu et al., 2018), WikiSQL (Zhong et al., 2017) and WTQ (Papernot et al., 2017), which are also free and open datasets for research use. For the table perturbation step, we hire professional annotators to find suitable RPL/ADD candidates for target columns. We pay the annotators at a price of 10 dollars per hour. The total time cost for annotating our benchmark is 253 hours. All the experiments in this paper can be run on a single Tesla V100 GPU. Our benchmark will be released along with the paper.
abstain abstain objective objective abstain method result other abstain abstain abstain abstain objective other abstain abstain abstain abstain method abstain abstain abstain abstain objective method abstain abstain abstain abstain method abstain abstain abstain abstain abstain objective objective objective objective method result result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain method abstain abstain abstain method method abstain abstain abstain abstain abstain abstain abstain abstain abstain other method abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain method abstain method abstain method abstain abstain method method abstain abstain abstain abstain method abstain method abstain abstain abstain abstain abstain abstain method method abstain abstain abstain method method method method abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain result abstain other result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain objective abstain objective abstain abstain method method abstain abstain abstain abstain abstain