We describe a span-level supervised attention loss that improves compositional generalization in semantic parsers. Our approach builds on existing losses that encourage attention maps in neural sequence-to-sequence models to imitate the output of classical word alignment algorithms. Where past work has used word-level alignments, we focus on spans; borrowing ideas from phrase-based machine translation, we align subtrees in semantic parses to spans of input sentences, and encourage neural attention mechanisms to mimic these alignments. This method improves the performance of transformers, RNNs, and structured decoders on three benchmarks of compositional generalization. Semantic parsers translate natural language utterances ( e.g., Schedule a meeting with Jean ) into executable programs ( e.g., CreateEvent( attendees=Jean) ), and play a crucial role in applications such as question answering systems and conversational agents (Liang, 2016; Gupta et al., 2018; Wen et al., 2017). As in many language understanding problems, a central challenge in semantic parsing is compositional generalization (Finegan-Dollak et al., 2018; Keysers et al., 2020). Consider a personal digital assistant for which developers have assembled separate collections of annotated utterances for user requests involving their calendars ( e.g., Schedule a meeting with Jean ) and their contact books ( e.g., Who is Jean's manager? ). An effective model should learn from this data how to additionally handle requests like Schedule a meeting with Jean's manager , composing skills from the calendar and contacts domains, with little or no supervision for such combinations. parsers (Dong and Lapata, 2016; Yin and Neubig, 2017), tend to perform poorly at out-of-distribution generalization of this kind (Lake and Baroni, 2018; Furrer et al., 2020; Suhr et al., 2020). Methods have been proposed to bridge the generalization gap using meta-learning (Lake, 2019; Wang et al., 2020) or specialized model architectures (Russin et al., 2019; Li et al., 2019; Liu et al., 2020; Chen et al., 2020). These have registered impressive performance on small synthetic benchmark datasets, but it has proven difficult to effectively combine them with large-scale pre-training (Lewis et al., 2020; Raffel et al., 2020) and natural data (Furrer et al., 2020). In contrast to this extensive literature on data transformations and model architectures, the design of loss functions to encourage compositional generalization has been under-explored. This paper investigates attention supervision losses that encourage attention matrices in neural sequence models to resemble the output of word alignment algorithms (Liu et al. (2016); Mi et al. (2016); Arthur et al. (2016); Lyu and Titov (2018), inter alia ) as a source of inductive bias for compositional tasks. Previous work has found that aligning program tokens ( e.g., FindManager in Fig. 1) to natural language tokens ( manager ) improves model performance (Misra et al., 2018; Rabinovich et al., 2017; Goldman et al., 2018; Richardson et al., 2018; Herzig and Berant, 2020; Oren et al., 2020). However, the token-level alignments derived from off-the-shelf aligners are often noisy, and the correspondence between natural language and program tokens is not always a many-to-one map of the kind returned by standard alignment algorithms. On the other hand, programs also have explicit hierarchical structure, which could be useful to induce better attention regularizers (Wang et al., 2019). Here we investigate the use of span-level alignments , identifying sub-programs that should be predicted as a unit and aligning all tokens in a sub-program to a CreateEvent CreateEvent( start=DateTime(date=Wednesday,time=NumberPM(2)), attendees=FindManager(recipient=Jean)) start DateTime date Wednesday time NumberPM 2 attendees FindManager recipient Jean A d d m ee ti n g w it h Je a n  s m a n a g e r o n W e d n e s d a y a t 2 PM ?x0 directed_by ?x1 edited_by ?x1 ?x1 art_directed M1 gender female W h a t d i d M 1  s f e m a l e a r t d i r ec t o r d i r ec t a n d e d it Utterance Token Vectors [CLS] In which city did ... Piotr 's 2007 Bangkok 1st corresponding natural language span (Herzig and Berant, 2020). We present a simple algorithm to derive span-level alignments from token-level alignments. Our approach is compatible with multiple models (RNNs, transformers, and structured tree-based de-coders), pretrained or not. In experiments, span-based attention supervision consistently improves over token-level objectives, achieving strong results on three semantic parsing datasets featuring diverse formalisms and tests of generalization. Supervised Attention Existing token-level supervised attention approaches assume access to an alignment matrix A | u || z | with entries a i,j , where a i,j = 1 iff the i -th source (utterance) token u i is aligned to the j -th target (program) token z j . A | u || z | can be inferred using latent variable models (Brown et al., 1993; Och and Ney, 2003; Dyer et al., 2013). During training, when the decoder predicts a target token z j , supervised attention encourages the target-to-source attention distribution p att ( u i | z j ) to match the prior alignment distribution p prior ( u i | z j ) = a i,j (cid:80) k a k,j , which is normalized by the number of source tokens aligned to z j . We use a squared error loss (Liu et al., 2016): L sup _ att = 1 | z | | z | (cid:88) j =1 | u | (cid:88) i =1 (cid:0) p att ( u i | z j )  p prior ( u i | z j ) (cid:1) 2 . Neural Semantic Parsers A semantic parser maps a natural language (NL) utterance u to an executable program z . In this paper, we consider neural parsers using token-based attentive decoders, in which z is predicted as a sequence of consecutive tokens { z | z | j =1 } by attending to tokens in u = { u | u | i =1 } . Examples include sequence-to-sequence models based on recurrent networks (Dong and Lapata, 2016; Jia and Liang, 2016) or transformers (Vaswani et al., 2017; Raffel et al., 2020), as well as structured parsing methods that predict a program following its syntactic structure (Dong and Lapata (2018), see 3 for more details). Previous work has also used a cross entropy loss (Rabinovich et al., 2017; Oren et al., 2020). Sub-program-to-Span Alignment We present a simple heuristic algorithm to extract span-level alignments between programs and utterances from existing token-level results (Algo. 1). Fig. 1 illustrates example span-level alignments for two types of programs (LISP and simplified SPARQL ). Similarly to Dong and Lapata (2018), we assume each program can be decomposed into a top-level sketch and a set of sub-programs . 1 For the LISP expression in Fig. 1a, the sketch contains the top-level function call ( CreateEvent( ? , ? ) ) and subprograms are named arguments paired with values 1 Unlike D&L, we allow sub-programs to include nonconsecutive (and possibly overlapping) spans of program tokens, e.g., {?x0 {edited_by {?x1}} in Fig. 1b. We also permit non-disjoint sub-programs. ( attendees=FindManager . . . ). For the SPARQL expression in Fig. 1b, sketches include the query form ( e.g., SELECT DISTINCT ) and sub-programs hold individual subject-relation-object assertions ( e.g., ?x0 edited_by ?x1 ). 2 In this paper, we use these program decompositions to guide span-level alignment. The underlying intuition is that every token in a sketch or sub-program will get aligned to the same set of utterance tokens. Algo. 1 extracts such set of utterance spans aligned to a sub-program z s from the set T z s of NL tokens that are aligned to tokens in z s (line 3). We present two variants of this approach, depending on the properties of the dataset (3). In the first case (lines 5-6), similar to bilingual phrase extraction in machine translation (MT; Och, 2002), we create a single consecutive utterance span u m : n via the outer bound of the aligned utterance tokens in T z s ( e.g., Block 1, Fig. 1a). In the second variant (lines 8-9), we find internally contiguous utterance spans (subsequences) in T z s and align them to z s . For instance, the sub-program ( ?x1 art_directed M1 ) in Block 2 of Fig. 1b aligns to two utterance spans: M1 's and art director . While this case does not have an exact analog in MT, it is reminiscent of the model of Chiang (2005) which extracts translation rules with discontinuous phrase segments, and could be useful in capturing long-range alignments of utterance subsequences to sub-programs 2 As we explain in Appendix B, such program decomposition could be easily generated using off-the-shelf syntax analyzers provided by the programming language. as in Block 2 (Andreas et al., 2013). Span-level alignments for a sub-program are then generated by pairing its program spans z p : q (spans with consecutive program tokens) with all its aligned utterance spans (lines 11-12). Finally, we generate alignments for sketch spans in z by pairing them with any utterance tokens that have not yet been aligned to a sub-program (lines 13-14). Algo. 1 leverages the explicit hierarchical structures of programs to generate alignments between sub-programs and utterance spans. Such an idea of using structural information for alignment extraction has deep roots in statistical syntax-based MT, which leverages the syntactic structure of sentences to generate alignments between parse trees and NL constituents (Galley et al., 2004; Chiang, 2005; Liu et al., 2006). Our approach is also broadly related to lexicon induction models in semantic parsers based on probabilistic CCG grammars (Kwiatkowski et al., 2011) or other formalisms (Jones et al., 2012), which learn mapping rules between logical form templates and utterance tokens. We evaluate span-level supervised attention on three benchmarks of compositional generalization. SMCALFLOW Compositional Skills (SMCALFLOW-CS) is a new dataset created in this study based on the task-oriented dialogue corpus SMCALFLOW (Semantic Machines et al., 2020), featuring real-world human-generated utterances about calendar management. Like the motivating story in 1, we create training data for skills S involving event creation ( e.g., Schedule a meeting with Adam ) and organization structure ( e.g., Who's on Adam's team? ), while evaluating on examples C featuring compositional skills ( e.g., Add meeting with Adam and his team ). Utterances are annotated with LISP -style programs (Fig. 1a). Since zero-shot compositional generalization is highly non-trivial due to novel language patterns ( e.g., Adam and his team ) and program structures ( e.g., usage of List(  ) to specify multiple attendees) in compositional examples, we consider a few-shot learning scenario, where a handful of compositional examples are included in the training set. Readers are referred to Appendix A for details of dataset construction. challenging compositional generalization dataset of 130 K synthetic utterances with SPARQL queries (Fig. 1b). Training and evaluation splits are constructed such that they have different distributions of compositional structures, while the distributions of atomic language ( e.g., director ) and program ( e.g., film.director ) constructs remain similar (Keysers et al., 2020). ATIS Text-to-SQL is a dataset of 3,809 SQL-annotated utterances about flight querying ( e.g., Flights from Seattle to Austin. ). We follow Oren et al. (2020) and use the query split (Finegan-Dollak et al., 2018), where training and evaluation programs do not overlap at template level. Models We apply span-level supervised attention to strong neural models on each dataset. We evaluate two systems on SMCALFLOW-CS: BERT 2S EQ , a sequence-to-sequence model with a BERT encoder and an LSTM decoder using copy mechanism, and COARSE 2F INE (Dong and Lapata, 2018), which uses (a BERT encoder and) a structured decoder that factorizes the generation of a program into sketch and value predictions. On CFQ, we use T5BASE (Raffel et al., 2020), and apply attention supervision on all the cross-attention heads in the last decoder layer. For ATIS , we take the best system from Oren et al. (2020) that is tuned for better generalization on this dataset, which is a sequence-to-sequence model with an ELMO encoder and coverage-based attention mechanism (See et al., 2017). We extract word alignments using IBM Model 4 in GIZA++ (Och and Ney, 2003), and canonicalize programs ( e.g., remove parentheses) to improve alignment quality. To extract span-level alignments, we use consecutive alignments (Case 1) in Algo. 1 for SMCALFLOW-CS and ATIS , as those datasets feature simple one-to-one mapping between subprograms and utterance spans. For CFQ, we use nonconsecutive alignments (Case 2) to handle assertions aligned to disjoint NL spans (Fig. 1b). We apply Eq. (1) during model optimization using either the token and span level alignment matrix for token ( + TS ) and span ( + SS ) level supervised attention, respectively. See Appendix B for details. Tab. 1 lists the evaluation results on SMCALFLOWCS with varying numbers of compositional examples in the training set ( C train ). 3 We report accuracies on both the in-domain single-skill examples ( S ) as well as on the generalized compositional-skill examples ( C ). Both methods improve compositional generalization for BERT 2S EQ and COARSE 2F INE , while span-level supervised attention is more effective. Intuitively, span-level alignments could better capture the correspondence between sub-structures in utterances and programs, helping the parser to correctly predict such sub-programs in compositionally novel contexts by focusing on the corresponding utterance span. Interestingly, in such a low-resource learning scenario with only a handful of training compositional samples, span-level supervised attention offers more gains in extreme low-resource settings ( | C train | = 16 ), outperforming the base BERT 2S EQ model by 13% absolute ( 33 . 6% v.s. 46 . 8% for BERT 2S EQ ). Indeed, we found that more alignment-like attentions are associated with more accurate model predictions. For a BERT 2S EQ model with span-level supervision trained on | C train | = 64 , when predicting subprograms for the attendees argument ( e.g., attendees=FindManager(recipient=self) ) on compositional samples in C , the model achieves 86% sub-program accuracy if it assigns a time-step 3 We ran GIZA++ and extracted span-level alignments for each training split separately. average of at least 90% of its attention weights over the aligned utterance spans ( e.g., with my manager ) identified by Algo. 1. Otherwise, the accuracy drops to 70% (more in Appendix C.1). Moreover, supervised attention may be a suf-ficient substitute for structured model architectures in some cases. Despite the unstructured BERT 2S EQ model's generally inferior performance without supervised attention, it matches the accuracies of COARSE 2F INE when both models are trained with span-level supervision. 4 We also remark that span-based supervision maintains or improves performance on in-domain single-skill examples ( S ). For instance, the accuracy for BERT 2S EQ increases from 82 . 8% to 83 . 9% when | C train | = 16 . Next, on CFQ (Tab. 2), we report break-down results based on the syntactic types of questions: R ecursive questions with chained multi-hop relations ( e.g., u r : Was M1 influenced by a German writer? ), and C onjunctive ones with only conjunctions of entities and relations and without chained relations ( e.g., u c : Was M1 directed and edited by M2 and M3? ). While supervised attention is effective on recursive questions, it struggles on conjunctive ones. This may be because the model learns to attend to discontinuous utterance spans ( e.g.,  M1 directed  and  M2 and M3  in u c ) when predicting a relation ( e.g., directed_by ) in a conjunction, which could be more sensitive to alignment 4 We found that the sketch and sub-program decoders in COARSE 2F INE do not achieve their best DEV . accuracy at the same iteration during training, which could hurt performance in our few-short learning setting. errors. Additionally, utterance spans aligned to a sub-program in conjunctive questions are usually longer and more complex ( e.g., having multiple conjunctive entity mentions like Did M1 write M2, M3, M4, and M5? ), which might require more fine-grained supervision than uniformly treating every aligned utterance tokens equally as in Eq. (1). More analysis is in Appendix C.2. Finally, we present the results on the ATIS query splits in Tab. 3, where span-level supervision is comparable with token-level one, further improving upon an already-strong model that targets for compositional generalization (ELMO with coverage based attention). Interestingly, token-level supervised attention is slightly worse than the baseline model on the standard i.i.d. splits, while span-level supervision does not offer further improvements. Empirically we observe that the utterance-SQL alignments in ATIS are much noisier than other two datasets due to redundant structures in SQL queries ( e.g., Join statements with intermediary ta-bles), whose aligned NL constituents are often not well defined (See Appendix B for more details). This paper demonstrated the effectiveness of span-level supervised attention as a simple and flexible tool for improving neural sequence models in a diverse set of architectures and tests of generalization. Future work might explore applications to other prediction tasks and joint learning of alignments with sequence model parameters. We thank the Semantic Machines team and anonymous reviewers for their valuable feedbacks. Pengcheng Yin was supported in part by an IBM Ph.D. fellowship.
result method method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain other abstain method abstain abstain abstain abstain abstain method abstain method abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain method abstain abstain abstain method abstain abstain abstain method abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result result abstain abstain abstain abstain result abstain abstain abstain result abstain result abstain abstain abstain abstain abstain result abstain abstain result objective abstain other other