Semantic parsers map natural language utterances into meaning representations ( e.g. pro-grams). Such models are typically bottlenecked by the paucity of training data due to the laborious annotation efforts. Recent studies have performed zero-shot learning by synthesizing training examples of canonical utterances and programs from a grammar, and further paraphrasing these utterances to improve linguistic diversity. However, such synthetic examples cannot fully capture patterns in real data. In this paper we analyze zero-shot parsers through the lenses of the language and logical gaps (Herzig and Berant, 2019), which quantify the discrepancy of language and programmatic patterns between the synthetic canonical examples and real-world user-issued ones. We propose bridging these gaps using improved grammars, stronger paraphrasers, and efficient learning methods using canonical examples that most likely reflect real user intents. Our model achieves strong results on the SCHOLAR and GEO benchmarks with zero labeled data. 1 1 Introduction Semantic parsers translate natural language (NL) utterances into formal meaning representations. In particular, task-oriented semantic parsers map user-issued utterances ( e.g. Find papers in ACL ) into machine-executable programs ( e.g. a database query), play a key role in providing natural language interfaces to applications like conversational virtual assistants (Gupta et al., 2018; Andreas et al., 2020), robot instruction following (Artzi and Zettle-moyer, 2013; Fried et al., 2018), as well as querying databases (Li and Jagadish, 2014; Yu et al., 2018) or generating Python code (Yin and Neubig, 2017). cost (Berant et al., 2013). Thus, the field has explored alternative approaches using supervisions cheaper to acquire, such as the execution results (Clarke et al., 2010) or unlabeled utterances (Poon, 2013). In particular, the seminal OVERNIGHT approach (Wang et al., 2015) synthesizes parallel data by using a synchronous grammar to align programs and their canonical NL expressions ( e.g. Filter(paper,venue= ? )  papers in ? and acl  ACL ), then generating examples of compositional utterances ( e.g. Papers in ACL ) with programs ( e.g. Filter(paper,venue=acl) ). The synthesized utterances are paraphrased by annotators, a much easier task than writing programs. Recently, Xu et al. (2020b) build upon OVERNIGHT and develop a zero-shot semantic parser replacing the manual paraphrasing process with an automatic paraphrase generator (2). While promising, there are still several open challenges. First, such systems are not truly zero-shot  they still require labeled validation data ( e.g. to select the best checkpoint at training). Next, to ensure the quality and broad-coverage of synthetic canonical examples, existing models rely on heavily curated grammars ( e.g. with 800 production rules), which are cumbersome to maintain. More importantly, as suggested by Herzig and Berant (2019) who study OVERNIGHT models using manual paraphrases, such systems trained on synthetic samples suffer from fundamental mismatches between the distributions of the automatically generated examples and the natural ones issued by real users. Specifically, there are two types of gaps. First, there is a logical gap between the synthetic and real programs, as real utterances ( e.g. Paper coauthored by Peter and Jane ) may exhibit logic patterns outside of the domain of those covered by the grammar ( e.g. Paper by Jane ). The second is the language gap between the synthetic and real utterances, as paraphrased utterances ( e.g. u (cid:48) 1 in Fig. 1) still follow similar linguistic patterns as the canonical ones they are 1455 paraphrased from ( e.g. u 1 ), while user-issued utterances are more linguistically diverse ( e.g. u 2 ). In this paper we analyze zero-shot parsers through the lenses of language and logical gaps, and propose methods to close those gaps (3). Specifically, we attempt to bridge the language gap using stronger paraphrasers and more expressive grammars tailored to the domain-specific idiomatic language patterns. We replace the large grammars of previous work with a highly compact grammar with only 46 domain-general production rules, plus a small set of domain-specific productions to capture idiomatic language patterns ( e.g. u 2 in Fig. 1, 3.1.1). We demonstrate that models equipped with such a smaller but more expressive grammar catered to the domain could generate utterances with more idiomatic and diverse language styles. On the other hand, closing the logical gap is non-trivial, since canonical examples are generated by exhaustively enumerating all possible programs from the grammar up to a certain depth, and increasing the threshold to cover more complex real-world examples will lead to exponentially more canonical samples, the usage of which is computationally intractable. To tackle the exponentially exploding sample space, we propose an efficient sampling approach by retaining canonical samples that most likely appear in real data (3.1.2). Specifically, we approximate the likelihood of canonical examples using the probabilities of their utterances measured by pre-trained language models (LMs). This enables us to improve logical coverage of programs while maintaining a tractable number of highly-probable examples as training data. By bridging the language and logical gaps, our system achieves strong results on two datasets featuring realistic utterances (SCHOLAR and GEO ). Despite the fact that our model uses zero annotated data for training and validation, it outperforms other supervised methods like OVERNIGHT and GRANNO (Herzig and Berant, 2019) requiring manual annotation. Analysis shows that current models are far from perfect, suggesting logical gap still remains an issue, while stronger paraphrasers are needed to further close the language gap. Problem Definition Semantic parsers translate a user-issued NL utterance u into a machine-executable program z (Fig. 1). We consider a zero-shot learning setting without access to parallel data in the target domain. Instead, the system is trained on a collection of machine-synthesized examples. Overview Our system is inspired by the existing zero-shot semantic parser AUTOQA (Xu et al., 2020b). Fig. 1 illustrates our framework. Intuitively, we automatically create training examples with canonical utterances from a grammar, which are then paraphrased to increase diversity in language style. Specifically, there are two stages. First, a set of seed canonical examples (Fig. 1b) are generated from a synchronous grammar , which defines compositional rules of NL expressions to form utterances (Fig. 1a). Next, in the iterative training stage, a paraphrase generation model rewrites the canonical utterances to more natural and linguistically diverse alternatives (Fig. 1c). The paraphrased examples are then used to train a semantic parser. To mitigate noisy paraphrases, a filtering model, which is the parser trained on previous iterations, rejects paraphrases that are potentially incorrect. This step of paraphrasing and training could proceed for multiple iterations, with the parser trained on a dataset with growing diversity of language styles. 2 Synchronous Grammar Seed canonical examples are generated from a synchronous context free grammar (SCFG). Fig. 1a lists simplified production rules in the grammar. Intuitively, productions specify how utterances are composed from lower-level language constructs and domain lexicons. For instance, given a database entity alan_turing with a property citations , u 3 in Fig. 1 could be generated using r 1 . Productions could be applied recursively to derive more compositional utterances ( e.g. u 2 using r 2 , r 4 and r 6 ). Our SCFG is based on Herzig and Berant (2019), consisting of domain-general rules of generic logical operations ( e.g. superlative , r 3 ) and domain-specific lexicons of entity types and relations. Different from Xu et al. (2020b) which uses a complex grammar with 800 rules, we use a compact grammar with only 46 generic rules plus a handful of idiomatic productions (3.1.1) to capture domain-specific language patterns ( e.g. most recent in u 2 , c.f. , u 1 ). Given the grammar, examples are enumerated exhaustively up to a threshold of number of rule applications, yielding a large set of seed canonical 2 This process is similar to expert iteration in reinforcement learning (Anthony et al., 2017), where a model is iteratively re-trained on newly discovered action trajectories. Paraphrase Generation and Filtering The paraphrase generation model rewrites a canonical utterance u to more natural and diverse alternatives u (cid:48) . u (cid:48) is then paired with u 's program to create a new example. We finetune a BART model on the dataset by Krishna et al. (2020), which is a subset of the PARANMT corpus (Wieting and Gimpel, 2018) that contain lexically and syntactically diverse paraphrases. The model therefore learns to produce paraphrases with diverse linguistic patterns, which is essential for closing the language gap when paraphrasing from canonical utterances. To further improve the syntactic diversity of paraphrases from imperative utterances ( e.g. u 2 , Fig. 1), we apply forced decoding such that half of the generated paraphrases start with questions with WH-prefixes ( e.g. u 3 in Fig. 1). Refer to Appendix A for details. Still, some paraphrases are noisy or potentially vague ( in Fig. 1c). We follow Xu et al. (2020b) and use the parser trained on previous iter-3 SCFGs could not generate utterances with context-dependent rhetorical patterns such as anaphora. Our model could still handle simple domain-specific context-dependent patterns ( e.g. Paper by A and B , where A and B are different authors) by first generating all the canonical samples and then filtering those that violate the constraints. ations as the filtering model, and reject paraphrases for which the parser cannot predict their programs. Language and Logical Gaps The synthesis approach in 2 will yield a large set of paraphrased canonical data (denoted as D par ). However, as noted by Herzig and Berant (2019) (hereafter HB19), the synthetic examples cannot capture all the language and programmatic patterns of real-world natural examples from users (denoted as D nat ). There are two mismatches between D par and D nat . First, there is a logical gap between real programs in D nat and the synthetic ones in D par , which are exhaustively composed up to a certain compositional depth and therefore cannot capture more complex programs in D nat . Next, there is a language gap between paraphrased canonical utterances and real-world user-issued ones. Real utterances ( e.g. u 2 in Fig. 1, which is from D nat but can be modeled as a canonical sample later in 3.1.1) enjoy more lexical and syntactic diversity, while the auto-paraphrased ones ( e.g. u (cid:48) 1 ) are typically biased towards the clunky language style of their canonical source ( e.g. u 1 ). While we could increase diversity via iterative rounds of paraphras-1457 ing ( e.g. u 2 (cid:55) u (cid:48) 2 (cid:55) u (cid:48)(cid:48) 2 ), the paraphraser could still fail on canonical utterances that are not natural English sentences at all, like u 1 . To close language gaps, we augment the grammar with productions capturing domain-specific idiomatic language styles. Such productions compress the clunky canonical expressions ( e.g. u 1 in Fig. 1) to more succinct and natural alternatives ( e.g. u 2 ), inspired by prior studies on how human experts revise canonical utterances (Wang et al., 2015), as well as by studying samples in real data. Specifically, we focus on two language patterns: Non-compositional expressions for multi-hop relations Compositional canonical utterances typically feature chained multi-hop relations that are joined together ( e.g. Author that writes paper whose topic is NLP ), which can be compressed using more succinct phrases to denote the relation chain, where the intermediary pivoting entities ( e.g. paper ) are omitted ( e.g. Author that works on NLP ). The pattern is referred to as sub-lexical com-positionality in Wang et al. (2015) and used by annotators to compress verbose canonical utterances, while we model them using grammar rules. Refer to Appendix B for more details. Idiomatic Comparatives and Superlatives The general grammar in Fig. 1a uses canonical constructs for comparative ( e.g. smaller than ) and superlative ( e.g. largest ) utterances ( e.g. u 1 ), which is not ideal for entity types with special units ( e.g. time, length). We therefore create productions specifying idiomatic comparative and superlative expressions ( e.g. paper published before 2014 , and u 2 in Fig. 1). Sometimes, answering a superlative utterance requires reasoning with other pivoting entities. For instance, the relation in venue that X publish mostly in between authors and venues implicitly involves counting the papers that X publishes. For such cases, we create macro productions, with the NL phrase mapped to a program that captures the computation involving the pivoting entity (Appendix B). Discussion Our SCFG uses idiomatic productions that capture domain-specific language expressions, together with simple domain-general rules (Herzig and Berant, 2019) to combine those idiomatic constructs to form compositional utterances. As we show in 4, both the base and idiomatic grammar sets are relatively compact, and we resort to strong paraphrasers to further natural-ize synthetic utterances and bridge the language gap. In line with Su and Yan (2017) and Marzoev et al. (2020), we remark that such functionality driven grammar engineering to cover representative patterns in real data using a small set of curated production rules is more efficient and cost-effective than example-driven annotation in classical supervised learning of semantic parsers, which requires labeling a sufficient number of parallel samples to effectively train a data-hungry neural model over a variety of underlying meanings and surface language styles. Our approach is also orthogonal with the prior work Xu et al. (2020b), which uses large curated general-purpose grammars to attempt to model English syntax, while using weak domain-specific rules that are much easier to specify than our SCFG, but might not be as effective to capture idiomatic language patterns in the domain. On the other hand, grammar engineering can be potentially costly. Ideally, one could study representative samples from real data and come up with a small set of idiomatic productions in the above categories that are expressive enough for domains like GEO and SCHOLAR (4). Still, the exact the amount of effort this process takes remains difficult to estimate. We present more discussion in 5. 3.1.2 Naturalness-driven Data Selection To cover real programs in D nat with complex structures while tackling the exponential sample space, we propose an efficient approach to sub-sample a small set of examples from this space as seed canonical data D can (Fig. 1b) for paraphrasing. Our core idea is to only retain a set of examples (cid:104) u , z (cid:105) that most likely reflect the intents of real users. We use the probability p LM ( u ) measured by a language model to approximate the naturalness of canonical examples. 4 Specifically, given all canonical examples allowed by the grammar, we form buckets based on their derivation depth d . For each bucket D ( d ) can , we compute p LM ( u ) for its examples, and group the examples using program templates as the key ( e.g. u 1 and u 2 in Fig. 1 are grouped together). For each group, we find the example (cid:104) u  , z (cid:105) with the highest p LM ( u  ) , and discard other examples (cid:104) u , z (cid:105) if ln p LM ( u  )  ln p LM ( u ) > 4 We use the GPT-2 XL model (Radford et al., 2019). (  = 5 . 0 ), removing unlikely utterances from the group ( e.g. u 1 ). 5 Finally, we rank all groups in D ( d ) can based on p LM ( u  ) , and retain examples in the topK groups. This method offers trade-off between program coverage and efficiency and, more surprisingly, we show that using only 0 . 2%  1% top-ranked examples also results in significantly better final accuracy (4). Zero-shot learning is non-trivial without a high-quality validation set, as the model might overfit on the (paraphrased) canonical data, which is subject to language and logical mismatch. While existing methods (Xu et al., 2020b) circumvent the issue using real validation data, in this work we create validation sets from paraphrased examples, making our method truly labeled data-free. Specifically, we consider a two-stage procedure. First, we run the iterative paraphrasing algorithm (2) without validation, and then sample (cid:104) u , z (cid:105) from its output with a probability p ( u , z )  p LM ( u )  (  = 0 . 4 ), ensuring the resulting sampled set D val par is representative. Second, we restart training using D valpar for validation to find the best checkpoint. The paraphrase filtering model is also initialized with the parser trained in the first stage, which has higher precision and accepts more valid paraphrases. This is similar to iterative training of weakly-supervised semantic parsers (Dasigi et al., 2019), where the model searches for candidate programs for unlabeled utterances in multiple stages of learning. We evaluate our zero-shot parser on two datasets. SCHOLAR (Iyer et al., 2017) is a corpus of user-issued queries to an academic database (Fig. 1). We use the version from HB19 with programs represented in  -calculus logical forms. The sizes of the train/test splits are 577/211. Entities in utterances and programs ( e.g. Parsing paper in ACL ) are canonicalized to slots ( e.g. keyphrase0 , venue0 ), and are recovered before executing the programs. We found in the dataset by HB19, slots are paired with with random entities for execution ( e.g. keyphrase0 (cid:55) Optics ). Therefore reference programs are likely to execute to empty results, making metrics like answer accuracy more prone to false-positives. We fix all such examples in the dataset, as well as those with execution errors. GEO (Zelle and Mooney, 1996) is a classical dataset with queries about U.S. geography ( e.g. Which rivers run through states bordering California? ). Its database contains basic geographical entities like cities, states, and rivers. We also use the release from HB19, of size 596/278. Models and Configuration Our neural semantic parser uses a BERT Base encoder (Devlin et al., 2019) and an LSTM decoder with copy mechanism. The paraphraser is a BART Large model (Lewis et al., 2020). We use the same set of hyper-parameters for both datasets. Specifically, we synthesize canonical examples from the SCFG with a maximal program depth of 6 , and collect the topK ( K = 2 , 000 ) GPT-scored sample groups for each depth as the seed canonical data D can (3.1.2), with two rounds of iterative paraphrasing and training (2). The beam size for the paraphraser is 20. We create validation sets of size 2 , 000 following 3.2. Refer to Appendix C for more details. Note that our model only uses the natural examples in both datasets for evaluation purposes, and the training and validation splits are not used during learning. Measuring Language and Logical Gaps We measure the language mismatch between utterances in the paraphrased canonical ( D par ) and natural ( D nat ) data using perplexities of natural utterances in D nat given by a GPT-2 LM fine-tuned on D par . For logical gap, we follow HB19 and compute the coverage of natural programs z  D nat in D par . Metric We report denotation accuracy on the execution results of predicted programs. 6 We ran all experiments with five random restarts and report the mean and standard deviation. In experiments, we first compare our model with existing approaches using labeled data. Next, we analyze how our proposed methods close the lan-6 We use SEMPRE (Berant et al., 2013) to execute  -calculus logical forms in parallel. guage and logical gaps. Tab. 1 reports test accuracies of various systems on the test sets, as well as their form of supervision. Specifically, the supervised parser uses a standard parallel corpus D nat of real utterances annotated with programs. OVERNIGHT uses paraphrased synthetic examples D par like our model, but with manually written paraphrases. GRANNO uses unlabeled real utterances u nat  D nat , and manual paraphrase detection to pair u nat with the canonical examples D can . Our model outperforms existing approaches without using any labeled data, while GRANNO , the currently most cost-effective approach, still spends $155 in manual annotation (besides collecting real utterances) on the two datasets (Herzig and Berant (2019), HB19). This demonstrates that our zero-shot parser is a data-efficient and cost-effective paradigm to train semantic parsers for emerging domains. Still, our system falls behind supervised models trained on natural corpora D nat , due to language and logical gaps between D par and D nat . Next, we explore whether our proposed methods are effective at narrowing the gaps and improving accuracy. Since the validation splits of the two datasets are small ( < 100 ), we evaluate on the full training/validation splits (around 600 examples) to get more reliable results. More expressive grammars narrow language and logical gaps We capture domain-specific language patterns using idiomatic productions to close language mismatch (3.1.1). Tables 2 and 3 list the results when we gradually augment the grammar with different categories of idiomatic productions. More expressive grammars help close the language gap, as indicated by the decreasing perplexities. This is especially important for SCHOLAR , which has diverse NL expressions hard to infer from plain canonical utterances. For instance, it could be non-trivial to paraphrase canonical utterances with multi-hop ( e.g. Author that cites paper by X ) or superlative relations ( e.g. Topic of the most number of ACL paper ) to more idiomatic alternatives ( e.g.  Author that cites X , and  The most popular topic for ACL paper ), while directly including such patterns in the grammar ( + Multihop Rel. and + Superlative ) is helpful. We also remark that the number of idiomatic productions we created is fairly compact (See Appendix B for a complete list). 7 We are able to 7 The base grammar is adapted from HB19, which defines entity types, example entities and (synonyms of) relations in Grammar Size ACC . improve the accuracy by 11 % absolute with 26 rules on SCHOLAR , while achieving 8 % gain using only 13 idiomatic productions on the simpler GEO domain with fewer entity types and relations. Additionally, more expressive grammars also improve logical coverage. The last columns ( Logical Cov. ) of Tables 2 and 3 report the percentage of real programs that are covered by the seed canonical data before ( D can ) and after ( D par ) iterative paraphrasing. Intuitively, a single idiomatic production often captures compositional computations like multi-hop queries, allowing the grammar to generate more compositional programs under the same threshold on the derivation depth. Notably, with all the full grammar on SCHOLAR , the number of exhaustively generated examples with a depth of 6 is tripled ( 530 K (cid:55) 1 , 700 K ). Moreover, recall that the seed canonical dataset D can contains examples with highly-likely utterances under the LM (3.1.2). Therefore, examples created by idiomatic productions are more likely to be included in D can . However, this could also be counter-productive, as such examples could dominate D can , "crowding out" other useful examples with lower LM scores. This likely explains the slightly decreased logical coverage on GEO (Tab. 3), as more than 30% samples in the LM filtered D can include idiomatic multi-hop relations directly connecting geographic entities with their countries ( e.g.  City in US ), while such examples only account for  8% of real data. While the over-representation issue might not negatively impact accuracy, we leave generating more balanced synthetic data as important future work. Finally, we note that the logical coverage drops after paraphrasing ( D can v.s. D par in Tables 2 and 3). This is because for some samples in D can , the paraphrase filtering model rejects all their paraphrases. We provide further analysis later in 5. Do smaller logical gaps entail better performance? As in 3.1.2, to make learning tractable in face of the exponential space of canonical samples, the seed canonical data D can used in iterative paraphrasing only consists of topK highest-scoring examples under GPT-2 for each program depth. However, using a smaller cutoff threshold K might sacrifice logical coverage, as fewer examples are in D can . To investigate this trade-off, we report results with varying K in Tab. 12. Notably, with K = 1 , 000 and around 3 K seed canonical data D can (before iterative paraphrasing), D can already covers 88% and 80% natural programs on SCHOLAR and GEO , resp. This small portion of samples only account for 0 . 2% ( 1% ) of the full set of 1 . 7 M + ( 0 . 27 M ) canonical examples exhaustively generated from the grammar on SCHOLAR (GEO ). This demonstrates our data selection approach is effective in maintaining learning efficiency while closing the logical gap. More interestingly, while larger K further closes the logical gap, the accuracy might not improve accordingly. This is possibly because while the coverage of real programs increases, the percentage of such programs in paraphrased canonical data D par (numbers in parentheses) actually drops. Out of the remaining 90%+ samples in D par not covered in D nat , many have unnatural intents that real users are unlikely to issue ( e.g.  Number of titles of papers with the smallest citations , or  Mountain whose elevation is the length of Colorado River ). Such unlikely samples are potentially harmful to the model, causing worse language mismatch, as suggested by the increasing perplexity when K = 8 , 000 . Similar to HB19, we observe around one-third of samples in D can and D par are unlikely. As we later discuss in 5, such unlikely utterances often have noisier paraphrases, which hurts the quality of D par . Comparing Data Selection Methods Next, we compare our proposed canonical data selection approach using GPT-2 with several baselines (Tab. 5 Upper Half ). First, randomly choosing examples from each level of program depth instead of using the topK GPT-scored ones results is less effective with higher variance. Further simplifying the procedure without constraining on equal sample size across program depths leads to significantly worse results, due to the scarcity of likely examples with simpler programs in the resulting sample set. Impact of Validation Data We generate validation data from samples of the paraphrased data in an initial run (3.2). Tab. 5 ( Lower Half ) compares this strategy with a baseline approach, which randomly splits the seed canonical examples in D can into training and validation sets, and runs the iterative paraphrasing algorithm on the two sets in parallel, with paraphrases from both sets filtered by the filtering model. This approach underperforms, since some canonical samples with program patterns in the natural data D nat can be partitioned into the validation split, and not used for training. Impact of Paraphrasers We rely on strong paraphrasers to generate diverse utterances to close the language gap. Tab. 6 compares the system using our paraphraser and the one in Xu et al. (2020b). Both are based on BART , while ours is fine-tuned to encourage lexically and syntactically diverse outputs (Appendix A). We measure lexical diversity using token-level F 1 between the original and para-1461 Example 1 (Uncommon Concept) u 1 Venue of paper by author 0 and published in year 0 u (cid:48) 1 , 1 author 0 's paper, published in year 0 u (cid:48) 1 , 2 Where the paper was published by author 0 in year 0 ? u (cid:48) 1 , 3 Where the paper was published in year 0 by author 0 ? u  nat Where did author 0 publish in year 0 ? (Wrong Answer) Example 2 (Novel Language Pattern) u 2 Author of paper published in venue 0 and in year 0 u (cid:48) 2 , 1 Author of papers published in venue 0 in year 0 u (cid:48) 2 , 2 Who wrote a paper for venue 0 in year 0 u (cid:48) 2 , 3 Who wrote the venue 0 paper in year 0 u  nat venue 0 year 0 authors (Correct) Example 3 (Unnatural Canonical Utterance) u 3 Author of paper by author 0 u (cid:48) 3 , 1 Author of the paper written by author 0 u (cid:48) 3 , 2 Author of author 0 's paper u (cid:48) 3 , 3 Who wrote the paper author 0 wrote? u  nat Co-authors of author 0 (Wrong Answer) Example 4 (Unlikely Example) u 4 Paper in year 0 and whose author is not the most cited author u (cid:48) 4 , 1 A paper published in year 0 that isn't the most cited author u (cid:48) 4 , 2 What's not the most cited author in year 0 u (cid:48) 4 , 3 In year 0 , he was not the most cited author Table 7: Case Study on SCHOLAR . For syntactic divergence, we use Kendall's  (Lapata, 2006) to compute the ordinal correlation of u and u (cid:48) . Our paraphraser outputs more diverse paraphrases ( e.g. What is the biggest state in US? ) from the source ( e.g. State in US and that has the largest area ), as indicated by lower token-level overlaps and ordinal coefficients, comparing to the existing paraphraser ( e.g. The state in US with the largest surface area ). Still, our paraphraser is not perfect, as discussed next. Our parser still lags behind the fully supervised model (Tab. 1). To understand the remaining bottlenecks, we show representative examples in Tab. 7. Low Recall of Filter Model First, the recall of the paraphrase filtering model is low. The filtering model uses the parser trained on the paraphrased data generated in previous iterations. Since this model is less accurate, it can incorrectly reject valid paraphrases u (cid:48) ( in Tab. 7), especially when u (cid:48) uses a different sentence type ( e.g. questions) than the source ( e.g. statements). Empirically, we found the recall of the filtering model at the first iteration of the second-stage training (3.2) is only around 60% . This creates logical gaps, as paraphrases of examples in the seed canonical data D can could be rejected by the conservative filtering model, leaving no samples with the same programs in D par . Imperfect Paraphraser The imperfect paraphraser could generate semantically incorrect predictions ( e.g. u (cid:48) 1 , 1 ), especially when the source canonical utterance contains uncommon or poly-semic concepts ( e.g. venue in u 1 ), which tend to be ignored or interpreted as other entities ( e.g. sites ). Besides rare concepts, the paraphraser could also fail to rewrite canonical utterances using more idiomatic syntax, like changing the mentioning of a conference using prepositional phrases ( u 2 ) to compound nouns ( u  nat in Example 2). While the model might still correctly answer u  nat , u  nat 's perplexity is high, suggesting language mismatch. Unnatural Canonical Utterances While we have attempted to close the language gap by generating more idiomatic canonical utterances, some of them are still not natural enough for the paraphraser to rewrite. This is especially problematic for relations not covered by our idiomatic productions, such as the co-authorship relation in Example 3. While this issue could be mitigated using additional production rules, grammar engineering could still remain challenging, as elaborated later. Unlikely Examples Besides the unnatural canonical utterances with clunky surface expressions but are still logically plausible, D can also contains around 30% unlikely examples with both unnatural utterances and convoluted meanings that almost certainly will not appear in real data ( e.g. u 4 ). Similar to unnatural utterances, their paraphrases are also much noisier ( e.g. u (cid:48) 4 ,  ), with only around 30% paraphrasing accuracy, compared to 70% for the likely ones. The filtering model is also less effective on unlikely examples (false positives ). These noisy samples will eventually hurt performance of the parser. We leave modeling utterance naturalness as important future work. Cost of Grammar Engineering Our approach relies on an expressive SCFG to bridge the language and logical gaps between synthetic and real data. While we have attempted to standardize the process of grammar construction by designing idiomatic productions following a set of representative grammar categories, grammar engineering still remains a non-trivial task. One need to have a good sense of the idiomatic language patterns that would frequently appear in real-world data, which requires performing user study or access to sampled data. Encoding those language patterns as production rules could also take a reasonable 1462 amount of time, depending on various factors, such as the complexity of the target domain and the proficiency of the user in the grammar formalism (  -calculus) used by our system. Still, we remark that most of the curated productions have simple syntactic constructs (a single verb, preposition, or adjective phrase, more in Appendix B.2.2), and we are able to significantly improve the performance over the base grammar (Ta-bles 2 and 3) using a relatively compact idiomatic grammar (10  30 rules on two datasets). Additionally, considering that the size of those idiomatic rules is orders of magnitude smaller than the size of the annotated parallel examples in the original datasets (around 800), it is safe to assume that for users familiar with the grammar formalism, curat-ing such a small set of grammar rules for domains similar to SCHOLAR and GEO is more efficient than labeling parallel samples in the original datasets. For the latter task the user would have to consider other factors, such as the coverage of compositional logical form patterns and language expressions, while our system automatically synthesizes compositional samples with diverse language style by composing (idiomatic) productions and iterative paraphrasing. Moreover, the paraphrased canonical examples synthesized from a compact curated grammar could also be used to bootstrap the collection of high-quality parallel data. Finally, creation of grammar rules could potentially be simplified by defining them using natural language instead of logical forms, reminiscent of studies on naturalizing programs using canonical language (Wang et al., 2017; Shin et al., 2021; Herzig et al., 2021). To mitigate the paucity of labeled data, the field has explored various supervision signals. Specifically, weakly-supervised methods leverage the denotations of utterances as indirect supervision (Clarke et al., 2010; Krishnamurthy and Mitchell, 2012), with programs modeled as latent variables (Berant et al., 2013; Pasupat and Liang, 2015). Optimization is challenging due to the noisy binary reward of execution correctness (Agarwal et al., 2019), calling for better learning objectives (Guu et al., 2017; Wang et al., 2021a) or efficient search algorithms for latent programs (Krishnamurthy et al., 2017; Liang et al., 2017, 2018; Muhlgay et al., 2019). Next, semi-supervised models leverage extra unlabeled utterances, using techniques like self-training (Konstas et al., 2017) or generative models (Kocisk et al., 2016; Yin et al., 2018). As a step further, unsupervised methods only use unlabeled utterances (Cao et al., 2019), and leverage linguistic scaffolds ( e.g. dependency trees) to infer programs with similar structures (Poon, 2013). Like our model, such methods use lexicons to capture alignments between NL phrases and logical predicates (Goldwasser et al., 2011), while our method does not require real utterances. Finally, methods based on OVERNIGHT (Wang et al., 2015) synthesize parallel corpora from SCFGs (Cheng et al., 2019; Xu et al., 2020a) or neural sequence models (Guo et al., 2018), and attempt to bridge the gaps between canonical and real utterances via paraphrase detection (Herzig and Berant, 2019) and generation (Su and Yan, 2017; Shin et al., 2021; Wu et al., 2021), or representation learning (Marzoev et al., 2020). In this paper, we propose a zero-shot semantic parser that closes the language and logical gaps between synthetic and real data. on SCHOLAR and GEO , our system outperforms other annotation-efficient approaches with zero labeled data. There are several import avenues for future work. First, dedicated approaches to generate syntactically diverse paraphrases using latent variable models, such as Hosking and Lapata (2021) and Hosking et al. (2022), could potentially improve performance. Additionally, systematic comparison with AUTOQA could help elucidate the impact of grammar quality to zero-shot semantic parsing, although this was not covered in this study due to complexities in porting  -calculus logical forms to the specialized formalism in AUTOQA. Next, careful human studies to understand the amount of efforts required for grammar engineering would provide more insights on the practicality of our approach. Finally, generalizing our approach to domains with more complex schemas ( e.g. ATIS ) is an important direction, which traditionally relies on careful feature engineering to reduce the amount of annotated data (Poon, 2013). We are grateful to Jonathan Herzig for sharing the code for GRANNO , and Silei Xu and Giovanni Campagna for questions regarding AUTOQA. Pengcheng Yin was supported in part by an IBM Ph.D. fellowship. We thank our anonymous reviewers for their insightful comments.
abstain abstain abstain abstain method objective result abstain other abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain objective method method objective abstain objective method result result result abstain abstain method abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain method abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method method abstain abstain method abstain abstain method method abstain method abstain abstain abstain abstain method abstain abstain method method method abstain abstain abstain abstain abstain abstain method method abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain result abstain abstain abstain abstain other result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain other other abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain method abstain method result abstain method abstain abstain other other other other other method other objective result abstain abstain abstain method method other other other