Recent work in cross-lingual semantic parsing has successfully applied machine translation to localize parsers to new languages. However, these advances assume access to high-quality machine translation systems and word alignment tools. We remove these assumptions and study cross-lingual semantic parsing as a zero-shot problem, without parallel data (i.e., utterance-logical form pairs) for new languages. We propose a multi-task encoder-decoder model to transfer parsing knowledge to additional languages using only English-logical form paired data and in-domain natural language corpora in each new language. Our model encourages language-agnostic encodings by jointly optimizing for logical-form generation with auxiliary objectives designed for cross-lingual latent representation alignment. Our parser performs significantly above translation-based baselines and, in some cases, competes with the supervised upper-bound. 1 1 Introduction Executable semantic parsing maps a natural language utterance to a logical form (LF) for execution in some knowledge base to return a denotation . The parsing task renders an utterance as a semantically identical, but machine-interpretable, expression grounded in a denotation. The transduction between natural and formal languages has allowed semantic parsers to become critical infrastructure in building human-computer interfaces for question answering, (Berant et al., 2013; Liang, 2016; Kollar et al., 2018), dialog systems (Artzi and Zettlemoyer, 2011), and robotics (Dukes, 2014). Recent advances in semantic parsing have improved accuracy for neural parsers (Jia and Liang, 2016; Dong and Lapata, 2016; Wang et al., 2020a) and examined their generalization capabilities with new dataset challenges (Zhong et al., 2017; Yu 1 Our code and data are available at github.com/tomsherborne/zx-parse . et al., 2018), in addition to considering languages other than English (Duong et al., 2017; inter alia. ). Prior work largely assumes that utterance-logical form training data is parallel in all languages (Jie and Lu, 2014), or must be created with human translation (Susanto and Lu, 2017a). This entry barrier to localization for new languages has motivated the exploration of machine translation (MT) as an economical alternative (Sherborne et al., 2020; Morad-shahi et al., 2020). However, MT can introduce performance-limiting artifacts and struggle to accurately model native speakers (Riley et al., 2020). Additionally, high-quality machine translation is less viable for lower resource languages, further limiting the appeal of MT-based approaches. In this work, we propose a new approach for zero-shot executable semantic parsing . Our method maximizes the success of cross-lingual transfer for a parser, trained on English paired data ( EN  LF ), to accurately generate logical forms 4134 from new languages ( X  LF ). Our goal is to parse utterances in a new language, l , without observing paired training data for this language, suitable machine translation, or bilingual dictionaries between l and English. Our critical dependencies are a pre-trained language model and utterance-logical form paired data for a source language (i.e., English). Aside from the zero-shot problem which is hard on its own (since paired data is not available for new languages), our semantic parsing challenge is further compounded with the difficulties inherent to structured prediction and the deficiency of copying strategies without gold token-level alignment (Zhu et al., 2020). We conceptualize cross-lingual semantic parsing as a latent representation alignment problem. As illustrated in Figure 1, we wish to encode different languages to an overlapping latent space for the decoder to have any chance at generating accurate logical forms. To achieve this, we train a decoder, conditioned upon encodings from a source language (e.g., English), to generate logical forms and simultaneously train encodings of a new language (e.g., Chinese) to be maximally similar to English. We hypothesize that if latent representations are aligned from a language-agnostic encoder, one can generate accurate logical forms from a new language without semantic parsing training data and thus eliminate the errors outlined in Figure 1. Our approach adopts a multi-task learning paradigm and trains a parser with auxiliary objectives, optimized to converge representations of additional new languages. We encourage language-agnostic representations by jointly optimizing for generating logical forms, reconstructing natural language, and promoting language invariance. Our intuition is that auxiliary losses can be exploited to induce similarity in a multi-lingual latent space. The effect of such alignment is that a decoder, trained only on English, can recognize an encoding from another language and generate the relevant logical form. Similar multi-task approaches have been successful in spoken-language understanding (van der Goot et al., 2021), text simplification (Mallinson et al., 2020; Zhao et al., 2020b), dependency parsing (Ahmad et al., 2019b), and machine translation (Arivazhagan et al., 2019). This work, to our knowledge is the first attempt to devise auxiliary objectives for executable semantic parsing as a zero-shot task. Our framework and hypothesis are also sufficiently flexible for application in additional zero-shot sequence transduction tasks. Our motivation is to improve parsing for non-English languages with maximal resource effi-ciency and minimal external dependencies beyond native-speaker utterances. We, therefore, induce a shared multilingual space without resorting to machine translation (Sherborne et al., 2020; Morad-shahi et al., 2020) and argue that our approach is superior because it (a) nullifies the introduction of translation or word alignment errors and (b) scales to low-resource languages without reliable MT. Experimental results on Overnight (Wang et al., 2015; Sherborne et al., 2020) and a new executable version of MultiATIS++ show that our parser generates more accurate logical forms with a minimized cross-lingual transfer penalty from English to French (FR), Portuguese (PT), Spanish (ES), German (DE), Chinese (ZH), Hindi (HI), and Turkish (TR). Cross-lingual Modeling This area has recently gained increased interest across several natural language understanding settings (Zhao et al., 2020a; Nooralahzadeh et al., 2020) with benchmarks such as XGLUE (Liang et al., 2020) and XTREME (Hu et al., 2020) allowing to study classification and generation tasks for multiple languages. Crosslingual approaches have also been developed for dependency parsing (Tiedemann et al., 2014; Schuster et al., 2019), sentence simplification (Mallinson et al., 2020), and spoken-language understanding (SLU; He et al., 2013; Upadhyay et al., 2018). Pre-training has shown to be widely beneficial for a wide range of cross-lingual models (Devlin et al., 2019; Conneau et al., 2020a). By virtue of being trained on massive corpora, these models purportedly learn an overlapping cross-lingual latent space (Conneau et al., 2020b) but have also been identified as under-trained for some tasks (Li et al., 2021), shown poor zero-shot performance, especially for languages dissimilar to English (Pires et al., 2019), and high variance (Keung et al., 2020). Semantic Parsing Most previous work (Lu, 2014; Susanto and Lu, 2017b,a) has focused on multilingual semantic parsing, i.e., learning from multiple natural languages in parallel, largely af-firming the benefit of high-resource multilingual data and multi-language ensemble training (Jie and Lu, 2014). Shao et al. (2020) further improved cross-lingual similarity with adversarial language 4135 identification across such ensembled training data. Code-switching in multilingual parsing has also been explored through mixed-language training datasets (Duong et al., 2017; Einolghozati et al., 2021). To adapt a parser to new languages, machine translation has been used as a reasonable proxy for in-language data (Sherborne et al., 2020; Morad-shahi et al., 2020). However, machine translation, in either direction can introduce limiting artifacts (Artetxe et al., 2020) with poor generalization due to how translationese training data diverges from gold test utterances (Riley et al., 2020). Zero-shot parsing has primarily focused on cross-domain' challenges to improve generalization across varying query structures and lexicons (Herzig and Berant, 2018; Givoli and Reichart, 2019) or different databases (Zhong et al., 2020; Suhr et al., 2020; Yu et al., 2018). The combination of zero-shot parsing with cross-lingual modeling has also been examined for the UCCA formalism (Hershcovich et al., 2019) and for task-oriented dialogue systems (see below). Dialog Modeling Cross-lingual transfer has been studied in the context of goal-oriented dialog for the spoken language understanding (SLU) tasks of intent classification and slot labeling (i.e., parsing an utterance into a semantic frame identifying the user's intent and its arguments). Recently released multilingual datasets like MultiATIS++ (Xu et al., 2020) and MTOP (Li et al., 2021) have facilitated the study of zero-shot transfer through the combination of pre-training, machine translation, and word alignment (to project annotations between languages). Recent work in this setting (Zhu et al., 2020; Li et al., 2021; Krishnan et al., 2021; Nicosia et al., 2021) identifies a penalty for cross-lingual transfer that neither pre-training nor machine translation can fully overcome. The primary challenge for cross-lingual parsing is learning parameters that can parse an utterance, x , from an unseen test language to an accurate logical form (LF). Typically, a parser trained on language l , or multiple languages { l 1 , . . . , l N } , is only capable for these languages and performs poorly outside this set. For a new language, prior approaches require parallel datasets and models (Jie and Lu, 2014; Haas and Riezler, 2016; Duong et al., 2017). In our work, zero-shot parsing refers to parsing utterances in new languages without paired data during training , For some language, l , there exists no pairing of x l to a logical form, y , except for English. 2 This setting also excludes silver-standard training pairs created using machine-translation. As these models have ultimately observed some form of utterance-logical form pairs for each new language, we do not consider such approaches here and refer to Sherborne et al. (2020) as an example of using MT for this task. It might be tempting to approach this problem as a case of fine-tuning a pre-trained (English) decoder for LF generation. Problematically, the output target is expressed in a formally defined language (e.g., SQL or   DCS) which models the semantics of questions very differently to natural language (e.g., without presumption or co-operation; Kaplan 1978). Formal languages (Kamp and Reyle, 1993) additionally present artifacts which render fine-tuning challenging such as unfamiliar syntax (e.g., table aliases or explicit recursion) and long output sequences. In practice, we observed fine-tuning leads to poor performance (e.g., < 1% accuracy on all languages), with the model insisting on hallucinating natural language. This is seemingly at odds with adjacent work in dialog modeling, which has found pre-trained decoders to be beneficial (Li et al., 2021). However, SLU requires learning a lightweight label vocabulary compared to the 200+ tokens required in LFs. Additionally, SLU typically maintains output sequences of similar size to natural language inputs (with tightly coupled syntactic compositionality between the two), whereas the syntactic and structural demands of LF generation are largely divorced from the input utterance. In our solution, the model is trained to parse from utterance-logical forms pairs only in English. Other languages are incorporated using auxiliary objectives and data detailed in Section 4. We explore the hypothesis that an overlapping multi-lingual latent space can be learned through auxiliary objectives in tandem with logical form generation (see Figure 2). Our intuition is that introducing these additional losses minimizes cross-lingual variance in latent encoding space by optimizing for language-agnostic representations with high similarity to the source language (i.e., English). Our approach minimizes the cross-lingual transfer penalty such that the zero-shot parser predicts logical forms from test inputs regardless of utterance language. By framing the cross-lingual parsing task as a latent representation alignment challenge, we explore a possible upper bound of parsing accuracy without errors from external dependencies. Section 6 demonstrates that our zero-shot model, using only English paired data and a small additional corpus, can generate accurate logical forms above translation baselines to compete with fully supervised in-language training. We adopt a multi-task sequence-to-sequence model (Luong et al., 2016) which combines logical form generation with two auxiliary objectives. The first is a language identification discriminator and the second is a reconstruction or translation decoder. An overview of our semantic parser is given in Figure 2; we describe each component below. Generating Logical Forms Predicting logical forms is the primary objective for our model. Given an utterance x = ( x 1 , x 2 , . . . , x T ) , we wish to generate logical form y = ( y 1 , y 2 , . . . , y M ) representing the same meaning in a machine-executable language. We model this transduction task using an encoder-decoder neural network (Sutskever et al., 2014) based upon the Transformer architecture (Vaswani et al., 2017). The sequence x is encoded to a latent representation z = ( z 1 , z 2 , . . . , z T ) through Equation (1) using a stacked self-attention Transformer encoder, E , with weights  E . z = E ( x |  E ) (1) p ( y | x ) = M (cid:89) i =0 p ( y i | y <i , x ) (2) p ( y i | y <i , x ) = soft ( DLF ( y <i | z,  DLF )) (3) LLF =  (cid:88) ( x, y ) S LF log p ( y | x ) (4) The conditional probability of the output sequence y is expressed in Equation (2) as each token y i is autoregressively generated based upon z and prior outputs, y <i . Equation (3) models distribution p ( y i | y <i , x ) using a Transformer decoder for logical forms, DLF , with associated weights  DLF where soft is the softmax function. We predict an output,  y , for semantic parsing dataset SLF = { x n , y n } Nn =0 , through the encoder and logical form decoder, { E, DLF } . Equation (4) describes the loss objective minimizing the cross-entropy between y and  y . Language Prediction Our first additional objective encourages language-agnostic representations by reducing the discriminability of the source language, l , from z . Equation (5) defines a L anguage P rediction (LP) network to predict l from z using a linear classifier over L training languages: LP ( x ) = W i x + b i (5) where W i  RL | z | and b i  RL are a weight and bias respectively. We follow the best model from Ahmad et al. (2019b). Equation (6) describes the conditional model for the output distribution where a language label is predicted using the time-average of the input encoding z of length T : p ( l | x ) = soft (cid:32) LP (cid:32) 1 T (cid:88) t z t (cid:33)(cid:33) (6) Finally, Equation (7) describes the objective function for the LP network: LLP =  (cid:88) x log p ( l | x ) (7) However, we reverse this gradient in the backward pass before the LP network, to encourage the encoder to produce language invariant representations (Ganin et al., 2016). The LP network is optimized to discriminate the source language from z , but the encoder is now optimized adversarially against this 4137 objective. Our intuition is that discouraging language discriminability in z encourages latent representation similarity across languages, and therefore reduces the penalty for cross-lingual transfer. Generating Natural Language The final objective acts towards both regularization and crosslingual similarity. Motivated by domain-adaptive pre-training (Gururangan et al., 2020), we further adapt the encoder towards question-style utterances from native speakers of each test language lacking task-specific training data. We add an additional Transformer decoder optimized to reconstruct a noisy input from latent representation z , in Equation (1). Utterance, x , is input to the encoder, E , and a separate decoder, DNL , then reconstructs x from z . We follow the denoising objective from Lewis et al. (2020) and replace x with noised input  x = N ( x ) with noising function N . The output probability of reconstruction is given in Equation (9) with each token predicted through Equation (10) using decoder, DNL , with weights  DNL :  z = E ( x |  E ) (8) p ( x |  x ) = T (cid:89) i =0 p ( x i | x <i ,  x ) (9) p ( x i | x <i ,  x )=soft ( DNL ( x <i |  z,  DNL )) (10) The auxiliary objectives are trained using both the utterances from SLF and monolingual data, SNL = {{ x n } Nn =0 } Ll =0 , in L languages (see Section 5). Submodel, { E, DNL } , predicts the reconstruction of x from  x with the following objective: LNL =  (cid:88) x log p ( x |  x ) (11) In the form described above, this objective requires only unlabeled, monolingual utterances in each target language. However, we can also augment it with a translation component to exploit natural language bi-text between the new language and English (e.g., SNL = {{ x n EN , x nl } Nn =0 } Ll =0 ) to further promote cross-lingual similarity. According to some sampling factor  , we randomly choose whether to reconstruct an utterance (as above) or translate to the parallel English utterance (i.e., replace x in Equation (11) with x EN ). training, an English query is encoded and input to all three objectives to express output loss as LLF + LNL + LLP . For new languages without ( x, y ) pairs, the utterance is encoded and input only to the auxiliary objectives for a combined loss as LNL + LLP . During inference, an utterance is encoded and always input to DLF to predict a logical form,  y , regardless of test language, l . During the backward pass, each output loss back-propagates the gradient signal from the respective objective function. For the encoder, these signals are combined as:  L  E =  LLF  E   LP  LLP  E +  NL  LNL  E (12)  = 2 1 + e  p  1 (13) where  { LP , NL } are loss weightings for auxiliary objectives and  is the reversed gradient scheduling parameter from Ganin et al. (2016). The  value increments with training progress p , scaled by  , according to Equation (13), to limit the impact of noisy predictions during early training. We expect that the parser will adapt and recognize an encoding from an unfamiliar language through our joint training process, and successfully connect new language representations to the logical-form decoder at test time. This sequence-to-sequence approach is highly flexible and may be useful for zero-shot approaches to additional generation tasks (e.g., paraphrasing). Semantic Parsing Datasets Our experiments examine whether our zero-shot approach generalizes across languages and domains. We evaluate performance on a new version of the ATIS dataset of travel queries (Hemphill et al., 1990; Dahl et al., 1994). We align existing English utterances and SQL logical forms from Iyer et al. (2017) to the multi-lingual utterances from the MultiATIS++ dataset for spoken language understanding (Xu et al., 2020). This alignment adds executable SQL queries to utterances in Chinese (ZH), German (DE), French (FR), Spanish (ES), and Portuguese (PT). We use the same 4,473/493/448 dataset split for training/validation/test as Kwiatkowski et al. (2011). We also add to the test set Hindi (HI) and Turkish (TR) utterances from Upadhyay et al. 4138 (2018). 3 We can now predict SQL from the ATIS test questions in eight natural languages. The Multi-ATIS++ Japanese set was excluded as the utterance alignment to this language was not recoverable. We also examine Overnight (Wang et al., 2015), an eight-domain dataset covering Basketball , Blocks , Calendar , Housing , Publications , Recipes , Restaurants , and Social Network domains. Overnight comprises 13,682 English utterances paired with   DCS logical forms, executable in SEMPRE (Berant et al., 2013), split into 8,754/2,188/2,740 for training/validation/test respectively. This training data exists only in EN and we use the ZH and DE test data from Sherborne et al. (2020) for multilingual evaluation. Given the varying linguistic phenomena across domains (e.g. relative spatial reasoning in Blocks or temporal arithmetic in Calendar ), this dataset presents a harder challenge for cross-lingual transfer. We measure performance with denotation accuracy as all inferred logical forms are executable in some knowledge base. This metric compares the retrieved denotation from the prediction,  y , to that from executing the gold-standard logical form. Dataset sizes are outlined in Appendix A. Natural Language Data For the reconstruction objective, we used the MKQA corpus (Longpre et al., 2020), a multi-lingual translation of 10,000 samples from NaturalQuestions (Kwiatkowski et al., 2019). This is suitable for our auxiliary objective as the utterances are native-speaker question surface forms, matching our test set while varying in subject. MKQA is also balanced across new languages to limit overexposure bias to one new language. For bi-text, we use the original English and the professionally translated question as a pair. We also report experiments using a sample of crawled data from ParaCrawl 7.1 (Ban et al., 2020). The sample comprises 10,000 web scraped sentences paired with equivalent English to form bi-text. Note that these samples are mostly declarative sentences and as such do not match the surface form of our test inputs (i.e., questions) and are also not parallel between sampled languages. We contrast this to MKQA to examine how the style of natural language data influences performance. For ATIS experiments, we use 60,000 utterances from each source in languages with training data (EN, FR, PT, ES, DE, ZH). For Overnight, we use 3 Misalignment between ATIS versions result in the test sets containing 442 and 381 utterances for HI and TR respectively. Model Configuration The implementation of ZX-PARSE (see Section 4) largely follows parameter settings from Liu et al. (2020) for Transformer encoder and decoder layers (see Appendix A for details on model configuration). ZX-PARSE requires an encoder model to generate multi-lingual latent representations for all objectives. Our main results use only the encoder component of mBART50 (Tang et al., 2020) and we present experiments using other pre-trained models in Appendix B. We use all pre-trained encoder layers and append one additional learnable layer. All decoders are randomly initialized six-layer stacks. Early experiments found this approach superior to any pretrained decoder initialization. The language predictor follows from Ahmad et al. (2019b) as a single linear classification layer mapping from 1,024 inputs to L output languages. Earlier findings supported that if the LP network is larger, then the reversed gradient signal is too strong and therefore less useful as the LP network can memorize the language. Comparison Models We primarily compare to a Translate-Test back-translation baseline wherein the new language test set is translated to English using Google Translate (Wu et al., 2016) and input to a reference sequence-to-sequence model trained on English. We also compare to Translate-Train, where we use MT from English to generate a proxy dataset in each new language (e.g., French, Portuguese, Spanish, German, Chinese, Hindi and Turkish) to train a monolingual parser. We consider improving upon these minimum effort baselines as a lower bound for justifying our approach. Additionally, we compare to an upper-bound monolingual model trained on professional translations of the new languages. We report results on MultiATIS++ for FR, PT, ES, DE, and ZH (profes-sional translations are not available for Overnight training data). This is the maximum effort strategy that we desire to avoid. Parameters for these reference systems match those outlined above e.g., mBART50 encoder to logical form decoder. Our results are outlined to answer four core questions, with additional ablations in Appendix B. Our findings support the hypothesis that we can minimize the cross-lingual transfer penalty by im-4139 proving latent alignment with auxiliary objectives. We also examine the latent space directly and find ZX-PARSE learns more similar representations between languages. Our parser achieves state-of-the-art zero-shot results for all non-English languages in the MultiATIS++ and Overnight benchmarks. Better than Translation? We compare between ZX-PARSE and the upperand lower-bounds in Table 1. Our multi-task approach significantly improves upon Translate-Test for all languages included within the auxiliary objectives ( p < 0 . 01 ). For ATIS, we find that Translate-Train performs below Translate-Test for languages similar to English (FR, ES, PT) but worse for more distant languages (DE, ZH). ZX-PARSE performance improves on Translate-Train for all languages included in reconstruction (EN, FR, PT, ES, DE, ZH), however, the general cross-lingual improvement in-sufficiently extends to additional languages (HI, TR) to perform above baselines. Within ZX-PARSE , French and German demonstrate the strongest zero-shot accuracy  +2 . 4% and +2 . 7% above the monolingual upper bound for ATIS. We do not observe similar improvement for Portuguese or Spanish despite their similarity to English. This may be a result of German and French dominating the pre-training corpora compared to other new languages. (Tang et al., 2020, their Table 6). Our model demonstrates similar significant improvement for Overnight ( p < 0 . 01 ), however, we find lesser gain compared to ATIS. This may be a consequence of the compounded challenge of evaluating eight varied domains of complex linguistic constructs. Here, we find that Translate-Train is a stronger approach than Translate-Test, which may be a consequence of machine-translation direction. Our best approach on German still improves above Translate-Train ( +4 . 0% ), however, we find performance on Chinese to be only marginally improved by comparison ( +0 . 6% ). We also observe some contrast in ZX-PARSE performance related to orthographic similarity to English. Parsing accuracy on Overnight in German is +6 . 2% above Chinese, with a similar +9 . 1% gap between these same languages for ATIS. Which Objective Matters? Ablations to the model are shown in Table 2, identifying the contributions of different objectives. Model (a) shows that without auxiliary objectives, performance in new languages is generally below Translate-Test. This is unsurprising, as this approach uses only pre-trained cross-lingual information without additional effort to improve similarity. Such efforts 4140 are incorporated in Model (b) using the additional reconstruction decoder. Even without the LP loss, domain targeted adaptation (with translation) improves cross-lingual parsing by an average across new languages of +9 . 3% for ATIS and +2 . 9% for Overnight. Notably, we identified an optimal ratio of translation to reconstruction of 50% (i.e.,  = 0 . 5 ). This suggests that both monolingual utterances (for domain-adaptive tuning) and bi-text (for translation) contribute to the utility of our method beyond reliance on one technique. Evaluating the LP objective within Model (c) and (d), we find the reversed gradient successfully reduces language discriminability. For Model (d), language prediction accuracy during training peaks at 93% after 2% progress and subsequently decreases to <8% beyond 10% of training. Language prediction accuracy for the test set is 7.2%. We observe a similar trend for Model (c). Comparing individual objectives, we find the addition of the language predictor alone less helpful than the reconstruction decoder. Comparing Model (a) and (c), we observe a smaller average improvement on new languages of +4 . 3% for ATIS and +1 . 8% for Overnight. This suggests adaptation towards specific surface form patterns can be more effective here than modeling languages as discrete labels. Considering the combination of objectives in Model (d), we identify cumulative benefit to parsing with both objectives. Compared to Model (a), the full model improves by an average of +16 . 3% for ATIS and +9 . 9% for Overnight across new languages. Our findings support our claim that latent cross-lingual similarity can be improved using auxiliary objectives and we specifically identify that a combination of approaches yields superior parsing. We suggest that this combination benefits from constructive interference, as the language prediction loss promotes invariance in tandem with multilingual generation tasks adapting the encoder to improve modeling the surface form (e.g., questions from native speakers) of the new language test data. Additional objectives also improve parsing for Hindi and Turkish despite neither being included within auxiliary training data (see HI and TR columns in Table 3). By adapting our latent representation to encourage similarity, we improve parsing accuracy for two typologically diverse languages without explicit guidance. To further examine this, we visualize the MultiATIS++ test set in Figure 3 and observe less discriminable encodings 80 60 40 20 0 20 40 60 80 60 40 20 0 20 40 60 mBART50 80 60 40 20 0 20 40 60 80 60 40 20 0 20 40 60 ZX-Parse EN FR PT ES DE ZH HI TR Figure 3: t-SNE comparison using mBART50 and ZX-PARSE encoders (MultiATIS++ test set). from ZX-PARSE compared to mBART50 . Quantitatively, we find the average cosine distance between the sentence-mean of parallel utterances reduces from 0.58 to 0.47. Similarly, the average token-level symmetric Hausdorff distance (Taha and Hanbury, 2015) between languages reduces from 0.72 to 0.41. This further supports that we learn more similar representations and our method has wider utility beyond explicitly targeted languages. Does Language Style Matter? In Table 3 we examine whether our auxiliary objectives are influ-enced by the style of natural language corpora for reconstruction. We find the use of questions positively improves performance compared to crawled sentences. Using questions either as monolingual utterances (i.e., no translation in DNL ) or with as a bi-text sample (i.e., reconstruction and translation in DNL ) improves above the Translate-Test baseline. We observe modest improvements with ParaCrawl, especially when introducing bi-text into DNL , but this is less consistent across languages. Overall, our results suggest that ZX-PARSE is robust even when question-style data is unavailable but can be particularly effective when adapting towards both new languages and domains. We also examined the influence of language family on performance (see Appendix B) and found that best performance utilizes a linguistically varied ensemble of languages. Omitting either Romance (ES/FR/PT) or Sino-Tibetan (ZH) languages in reconstruction negatively impacts performance. Where Does Improvement Come from? Comparing to Translate-Test, on ATIS, our best model generates 32% fewer ill-formed SQL requests and 24% fewer extraneous queries accessing unrelated tables in the database. Translation can fail when entities are mishandled and our model generates 36% fewer queries with erroneous named entities. For Overnight, gains are strongly related to improved numeracy in the model. Between our full 4141 ATIS Overnight Baselines EN FR PT ES DE ZH HI TR EN DE ZH Translate-Train  55.9 56.1 57.1 60.1 56.1 56.3 45.4  62.2 59.4 Translate-Test  58.2 57.3 57.9 56.9 51.4 52.6 52.7  60.1 48.1 ZX-PARSEMKQA  = 0 . model and simplest approach (Model (a) in Table 2), we find more well-formed logical forms account for the largest improvement (32.5% fewer ill-formed SQL queries for ATIS and 35.2% fewer ill-formed  -DCS queries for Overnight). This supports our notion in Figure 1 that better latent alignment can minimize cross-lingual penalty. However, improved structure prediction is insufficient to solve this task on its own; 58.7% of remaining errors in the best model are due to mishandled entities with the highest entity errors for Chinese (60.2%) and lowest for French (36.7%). This suggests that aligning entities across languages might be necessary for further improvement. We presented a multi-task model for zero-shot cross-lingual semantic parsing which combines logical form generation with auxiliary objectives that require only modest natural language corpora for localization. Through aligning latent representations, ZX-PARSE minimizes the error from cross-lingual transfer and improves accuracy across languages unseen during training. Although we focused exclusively on executable semantic parsing, our approach is general and potentially relevant for linguistically motivated frameworks such as Abstract Meaning Representation (Banarescu et al., 2013; Damonte and Cohen, 2018) or Discourse Representation Theory (Kamp and Reyle, 1993; Evang and Bos, 2016). In the future, we will investigate a few-shot scenario and study sample efficient cross-lingual transfer by explicitly promoting generalization using techniques such as meta-learning (Finn et al., 2017). A key limitation of our work is the limited coverage of eight higher-resource languages. As such, we are unable to test our approach in a genuinely low-resource scenario. We must also consider the risk of over-generalization to dominant dialects within each language as we lack an evaluation of additional dialects (e.g. our English dataset is representative of American English but not Indian English). We hope that such issues can be addressed with additional data collection. Our training requirements are detailed in Appendix A. We hope our work contributes to further usage and development of singular multilingual models as opposed to learning N monolingual models for N languages. We thank the anonymous reviewers for their feedback and Bailin Wang, Kate McCurdy, and Rui Zhang for insightful discussion. The authors gratefully acknowledge the support of the UK Engineering and Physical Sciences Research Council (grant EP/L016427/1; Sherborne) and the European Research Council (award number 681760; Lapata).
abstain abstain objective objective objective method abstain abstain abstain abstain abstain abstain abstain abstain abstain objective objective objective method abstain abstain method objective objective objective method abstain abstain abstain objective abstain objective abstain abstain other other other other other other other other other other other other other other other abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain method abstain abstain abstain abstain abstain abstain abstain method method abstain other abstain method abstain abstain abstain method other abstain method abstain abstain method abstain abstain other abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain method method method abstain method method method abstain method abstain method abstain method abstain method method abstain method method abstain method method method method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain result result abstain abstain abstain result abstain abstain abstain abstain result abstain abstain abstain abstain result result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result result abstain result abstain abstain abstain abstain result abstain result abstain result abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain objective abstain method abstain abstain abstain method abstain abstain other other