We present substructure distribution projection (SUBDP), a technique that projects a distribution over structures in one domain to another, by projecting substructure distributions separately. Models for the target domain can then be trained, using the projected distributions as soft silver labels. We evaluate SUBDP on zero-shot cross-lingual dependency parsing, taking dependency arcs as substructures: we project the predicted dependency arc distributions in the source language(s) to target language(s), and train a target language parser on the resulting distributions. Given an English treebank as the only source of human supervision, SUBDP achieves better unlabeled attachment score than all prior work on the Universal Dependencies v2.2 (Nivre et al., 2020) test set across eight diverse target languages, as well as the best labeled attachment score on six languages. In addition, SUBDP improves zero-shot cross-lingual dependency parsing with very few (e.g., 50) supervised bitext pairs, across a broader range of target languages. Zero-shot cross-lingual dependency parsing is the task that requires prediction of dependency parses without seeing any parsing example in the target language; instead, the model may use annotated parses in other languages. A popular line of work is annotation projection: the parses generated by a source language dependency parser are projected into the target language, where the projected parses are then used to train a new parser. As illustrated in Figure 1b, most annotation projection methods typically output partial hard dependency trees, 1 where there either is or is not an arc between any pair of 1 Throughout this paper, we refer to dependency parse trees with 0/1 arc and label probabilities, i.e., conventional dependency trees, as hard trees ; in contrast, we refer to collections of per-word head distributions and per-arc label distributions with continuous probabilities as soft trees . (b) Projection with only one-to-one alignments. words. In addition, most bitext-based work has relied on one-to-one word alignment between bitext pairs (e.g., I and  in Figure 1; Ma and Xia, 2014; Lacroix et al., 2016; Rasooli et al., 2021, inter alia ), discarding information in many-to-one alignments (e.g., book store and   in Figure 1). In this work, we introduce substructure distribution projection (SUBDP; Figure 1a), where dependency arcs act as substructures. We project substructure distributions, i.e., the conditional prob-6547 ability distribution of the corresponding head given a word. 2 When the source parse is a hard tree, SUBDP has the same behavior as prior work (e.g., Lacroix et al., 2016) for arcs that are only involved in one-to-one alignments; for many-to-one alignments, SUBDP projects the corresponding arcs into soft arc distributions in the target language. Therefore in SUBDP, a target language word may have multiple heads in the projected trees, where their probabilities sum to one. More generally, SUBDP may take dependency arc or label distributions (i.e., soft trees) in the source language(s), instead of hard trees, as the input. As in annotation projection approaches, the projected soft trees are then used to train a target language parser. We evaluate SUBDP on zero-shot cross-lingual dependency parsing with eight diverse languages from the Universal Dependencies v2.2 (Nivre et al., 2020), where the English treebank is the only source of human supervision. Taking English as the source language, SUBDP significantly outperforms all baseline methods on all distant languages (Arabic, Hindi, Korean, and Turkish) in our experiments, in terms of both labeled attachment scores (LAS) and unlabeled attachment scores (UAS), while achieving superior UAS on all nearby languages (German, French, Spanish, and Italian) as well. Further analysis shows that SUBDP also helps improve zero-shot cross-lingual dependency parsing with a small amount of supervised bitext, across a broader range of target languages. Zero-shot cross-lingual dependency parsing. 3 Existing approaches can be classified into the following categories: 1. Delexicalized training (Zeman and Resnik, 2008; McDonald et al., 2011; Cohen et al., 2011; Durrett et al., 2012; Rosa and abokrtsk, 2015, inter alia ), which only considers delexicalized features (e.g., part-of-speech tags) in training. 2. Transfer with cross-lingual embeddings (Tckstrm et al., 2012; Guo et al., 2015; Schuster et al., 2019, inter alia ), which assumes that cross-lingual word representations, including word clusters (Tckstrm et al., 2012; Ammar 2 Projection of the distribution over whole parse trees has been considered by Ma and Xia (2014), while SUBDP has a much lower time complexity  see 2 for more discussion. 3 Also referred to as zero-shot dependency parsing in recent literature (Schuster et al., 2019; Wang et al., 2019). et al., 2016), word type embeddings (Guo et al., 2015, 2016; Duong et al., 2015; Ammar et al., 2016; Wick et al., 2016), or contextualized crosslingual word embeddings (Schuster et al., 2019; Wang et al., 2019; He et al., 2019; Ahmad et al., 2019a,b), provide shared features for words with similar syntactic roles. 3. Treebank translation , which translates treebanks in the source language(s) into the target language(s) (Tiedemann et al., 2014; Tiedemann, 2015; Tiedemann and Agic, 2016) or a code-switching mode (Zhang et al., 2019), and uses them to train target language parsers. 4. Annotation projection , 4 which trains a parser in the source language(s), and projects the predicted source language parse trees to target language(s) using bitext (Hwa et al., 2005; Ma and Xia, 2014; Agic et al., 2016). Additional strategies are usually used to improve the projection quality, such as keeping confident edges only (Li et al., 2014; Lacroix et al., 2016), projection from multiple source languages (Tck-strm et al., 2013; Agic et al., 2016; Rasooli and Collins, 2017), density based iterative filtering (Rasooli and Collins, 2015, 2017, 2019), and noisy self-training (Kurniawan et al., 2021). These approaches make different assumptions on annotation availability, such as gold part-of-speech tags (Zeman and Resnik, 2008; Cohen et al., 2011; Durrett et al., 2012, inter alia ), a reasonably good translator, which uses extra annotated data in the training process (Tiedemann et al., 2014; Tiedemann, 2015; Zhang et al., 2019), high-quality bilingual lexicons (Durrett et al., 2012; Guo et al., 2015, 2016, inter alia ), or language-specific constraints (Meng et al., 2019). Most bitext-based work assumes annotated bitext (Ma and Xia, 2014; Li et al., 2014; Lacroix et al., 2016, inter alia ) or bitext constructed from extra signals (e.g., Wikipedia; Rasooli et al., 2021). However, He et al. (2019), Schuster et al. (2019), Ahmad et al. (2019a,b), and Kurniawan et al. (2021) only require minimal annotations (i.e., source language treebanks and unlimited raw text in relevant languages). We are mainly interested in the minimal annotation setting, and will compare to this line of work. Our proposed method, SUBDP, falls into the category of annotation projection. Some of the 4 We use annotation projection to denote the projection of predicted parses following Rasooli and Collins (2019) and Zhang et al. (2019), and treebank translation for the projection of human-annotated trees following Tiedemann et al. (2014). benefits of SUBDP relative to prior work are that it works well with minimal annotations, allows soft word alignment (3.2), supports both labeled and unlabeled parsing, and has a low time complexity O ( n 2 ) for non-projective parsing. 5 SUBDP can be easily extended to other tasks, such as sequence labeling, where we can define substructures (Shi et al., 2021a) and substructure distributions. Multilingual contextualized representations. Recent contextualized models pretrained on multilingual text (Devlin et al., 2019; Conneau et al., 2020; Tran et al., 2020, inter alia ) are effective across a wide range of cross-lingual NLP tasks, including bitext retrieval (Tran et al., 2020), bilingual lexicon induction (Shi et al., 2021b), cross-lingual named entity recognition (Pires et al., 2019; Mulcaire et al., 2019), and cross-lingual dependency parsing (Schuster et al., 2019; Wang et al., 2019). In this work, we apply two of the contextualized pretrained models, XLM-R (Conneau et al., 2020) and CRISS (Tran et al., 2020) to generate unsupervised bitext. Soft-label methods. Calculating the cross entropy loss between model output and a soft distribution (instead of one-hot labeles) has been applied to knowledge distillation (Hinton et al., 2015; You et al., 2017; Sanh et al., 2019, inter alia ), crosslingual named entity recognition (Wu et al., 2020), and for handling annotation discrepancy (Forna-ciari et al., 2021). Our approach is a type of soft-label method, with additional post processing to the output of the original models. Our pipeline for zero-shot cross-lingual dependency parsing consists of three steps: (1) train a bi-affine dependency parser P 1 in the source language L 1 , (2) project annotations on L 1 sentences to their parallel sentences in the target language L 2 (3.3), and (3) train another bi-affine dependency parser P 2 for L 2 (3.4). We first present some background (3.1) and preliminaries (3.2). parser. For a sentence with n words (cid:104) w 1 , . . . , w n (cid:105) , 6 we denote the word features when acting as heads and dependents by 5 In contrast, Ma and Xia (2014) require O ( n 4 ) time for non-projective unlabeled dependency parsing. H  (cid:82) n  d h and D  (cid:82) n  d d respectively, where d h and d d denote the dimensionality of the corresponding features. The probability of word w i having head w j can be formulated as an n -way classification problem: S (arc) = DW (arc) H (cid:124) (1) P ( w j | w i ) = exp (cid:16) S (arc) i,j (cid:17) (cid:80) nk =1 exp (cid:16) S (arc) i,k (cid:17) , (2) where W (arc)  (cid:82) d d  d h is the parameters of the bi-affine module. 7 Given log P ( w j | w i ) for every pair of i and j , the dependency trees can be inferred by finding the spanning arborescence of maximum weight using the ChuLiu-Edmonds algorithm (Chu and Liu, 1965; Edmonds, 1968). We use the algorithm proposed by Tarjan (1977), which has an O ( n 2 ) time complexity for each sentence. We denote the candidate dependency label set by L . Parameterized by W (label)  (cid:82) d d  d h | L | , we define the probability that the arc from head w j to dependent w i has the label (cid:96) by S (label) i,j,(cid:96) = (cid:88) p (cid:88) q D i,p W (label) p,q,(cid:96) H j,q P ( (cid:96) | w j  w i ) = exp (cid:16) S (label) i,j,(cid:96) (cid:17) (cid:80) | L | k =1 exp (cid:16) S (label) i,j,k (cid:17) , (3) Given the probability definitions above, the model is trained to maximize the log likelihood of the training data. More details can be found in Dozat and Manning (2017). We use bi-affine dependency parsers as the backbone for all parsers in this work, though it is worth noting that SUBDP works for any parser that produces a set of arc and label distributions. CRISS. CRISS (Tran et al., 2020) is an unsupervised machine translation model trained with monolingual corpora, starting from mBART (Liu et al., 2020), a multilingual pretrained sequence-to-sequence model with a mask-filling denoising objective. During the training process, CRISS iteratively (1) encodes sentences in the monolingual corpora with its encoder, (2) mines bitext based on encoding similarity, and (3) uses the mined bitext to fine-tune the model with a machine translation objective. In this work, we use CRISS to generate 7 While Eq (1) is in a bi-linear form, in practice, we can always append a constant feature column to both H and D , resulting in a bi-affine model. unsupervised translation of English sentences to construct bitext, and apply its encoder to extract word features for an ablation study. SimAlign. SimAlign (Jalili Sabet et al., 2020) is a similarity based word aligner: given a pair of source and target sentence (cid:104) s, t (cid:105) , SimAlign computes a contextualized representation for each token in both s and t using multilingual pretrained models (Devlin et al., 2019; Conneau et al., 2020), and calculates the similarity matrix S , where S i,j represents the cosine similarity between tokens s i and t j . The argmax inference algorithm selects position pairs (cid:104) i, j (cid:105) , where S i,j is both horizontal and vertical maximum, and outputs the word pairs corresponding to such position pairs as the word alignment. In this work, we use XLM-R (Conneau et al., 2020) based SimAlign with the argmax algorithm to extract word alignment for SUBDP. It is worth noting that pretrained multilingual models usually use byte-pair encodings (BPEs; Gage, 1994), a more fine-grained level than words, for tokenization. The argmax algorithm may therefore generate many-to-one alignments. More details can be found in Jalili Sabet et al. (2020). Unlike bitext based word alignment (Och and Ney, 2003; Dyer et al., 2013), SimAlign does not require any bitext to produce high quality alignments, and therefore better fits the low-resource scenario with very few bitext pairs available. Dependency annotations in L 1 . As in the most common data settings for supervised dependency parsing, we assume access to sentences with dependency annotations: for a sentence (cid:104) w 1 , . . . , w n (cid:105) , there is a dummy word w 1 , whose unique dependent is the root word; every other word w i is labeled with h i and r i , denoting that the head of w i is w h i , with the dependency relation r i . We use these annotations to train an L 1 bi-affine dependency parser P 1 , following the procedure described in 3.1. Bitext. We denote the available m pairs of bitext by B = {(cid:104) s ( k ) , t ( k ) (cid:105)} mk =1 , where { s ( k ) } and { t ( k ) } are sentences in L 1 and L 2 respectively. Word alignment. For a bitext pair (cid:104) s, t (cid:105) , we generate the word alignment matrix  A  { 0 , 1 } | s || t | with SimAlign, where  A i,j = 1 denotes that there exists an alignment between s i and t j . We would like the word alignment matrices to be right stochastic, i.e., (1) each element is nonnegative and (2) each row sums to one, to ensure that the results after projection remain distributions. To handle words that have zero or more than one aligned words in the other language, we introduce the following two matrix operators. The add-dummy-position operator (  ) :  : (cid:82) r  c  (cid:82) ( r +1)  ( c +1) (  r, c  (cid:78) + ) ( M ) i,j = M i,j (1  i  r, 1  j  c ); ( M ) i,c +1 = (cid:48) [ M i, 1 , . . . , M i,c ](1  i  r ); ( M ) r +1 ,j = 0(1  j  c ); ( M ) r +1 ,c +1 = 1 , where (cid:48) [  ] = 1 when all input values are zero and otherwise 0 . The row normalization operator NR (  ) : NR : (cid:82) r  c  (cid:82) r  c (  r, c  (cid:78) + ) NR ( M ) i,j = M i,j (cid:80) (cid:96) M i,(cid:96) . Intuitively, the added dummy positions correspond to null words in the word alignment literature (Dyer et al., 2013; Schulz et al., 2016; Jalili Sabet et al., 2020, inter alia ). We denote the source-to-target alignment matrix by A s  t = NR (cid:16) (  A ) (cid:17) , and the target-to-source alignment matrix by A t  s = NR (cid:16) (  A (cid:124) ) (cid:17) . stochastic matrices by definition. Arc distribution projection. We consider a pair of bitext (cid:104) s, t (cid:105) . Let P 1 ( s j | s i ) denote the arc probability produced by the parser P 1 . Like the dummy position notation, we specify a dummy ( | s | + 1) th word whose head is itself, that is, P 1 ( s i | s | s | +1 ) = 0 , P 1 ( s | s | +1 | s | s | +1 ) = 1 . We project P 1 (  |  ) to  P 2 ( t q | t p ) , the arc probability distributions in the parallel L 2 example t ,  P 2 ( t q | t p )= | s | +1 (cid:88) i =1 | s | +1 (cid:88) j =1 A t  s p,i P 1 ( s j | s i ) A s  t j,q . (4) It is guaranteed that  P 2 (  | t p ) is a distribution for any t p  a proof can be found in Appendix A.1. Note that if we adopt matrix notations, where we denote  P 2 ( t q | t p ) by  P (2) p,q and denote P 1 ( s j | s i ) by P (1) i,j , Eq (4) is equivalent to  P (2) = A t  s P (1) A s  t . Label distribution projection. Let P 1 ( (cid:96) | s j  s i ) denote the label probability produced by P 1 . For dummy positions, we simply add a uniform distribution, that is, P 1 ( (cid:96) | s j  s i ) = 1 L if i or j = | s | + 1 . We project P 1 ( | ) to  P 2 ( (cid:96) | t q  t p ) , the label distributions in the parallel L 2 example t , by  P 2 ( (cid:96) | t q  t p )= | s | +1 (cid:88) i =1 | s | +1 (cid:88) j =1 A t  s p,i P 1 ( (cid:96) | s j  s i ) A t  s q,j  P 2 (  | t q  t p ) is provably a distribution for any pair of t p and t q (see Appendix A.2). We train another bi-affine dependency parser P 2 on language L 2 , by minimizing the cross entropy between its produced probability P 2 and the soft silver labels  P 2 . Note that the added dummy word denoting the null alignment is not eventually used in the final dependency inference process and may introduce extra noise to the model, so we instead calculate the partial cross entropy loss, which does not consider elements involving dummy words. Concretely, we compute the partial arc cross entropy loss for one example t as follows: L ( t ) arc ( P 2 ,  P 2 )=  | t | (cid:88) p =1 | t | (cid:88) q =1  P 2 ( t q | t p ) log P 2 ( t q | t p ) Similarly, the partial label cross entropy loss can be computed as follows: L ( t ) label ( P 2 ,  P 2 ) =  | L | (cid:88) (cid:96) =1 | t | (cid:88) p =1 | t | (cid:88) q =1  P 2 ( (cid:96) | t q  t p ) log P 2 ( (cid:96) | t q  t p ) Finally, we train the parameters of P 2 to minimize (cid:88) (cid:104) s,t (cid:105)B L ( t ) arc ( P 2 ,  P 2 ) + L ( t ) label ( P 2 ,  P 2 ) . Throughout all experiments, the subword representation is a weighted sum of layer-wise representation from a frozen pretrained model, where each layer has a scalar weight optimized together with other network parameters to minimize Eq. (5). We convert subword features to word features by endpoint concatenation, following Toshniwal et al. (2020). We use the Adam optimizer (Kingma and Ba, 2015) to train all models, where the source language parser is trained for 100 epochs with initial learning rate 2  10  3 following the baseline implementation by Zhang et al. (2020), and the target language parser is trained for 30 epochs with initial learning rate 5  10  4 . 8 We use the loss against silver projected distributions on the development set for SUBDP and the development LAS against projected trees for baselines for early stopping. 9 For evaluation, we ignore all punctuation following the most common convention (Ma and Xia, 2014; Rasooli and Collins, 2015; Kurniawan et al., 2021, inter alia ). If not specified,  All models in target languages are initialized with the trained source language parser. All word alignments are obtained by XLM-R based SimAlign (Jalili Sabet et al., 2020), using BPE tokenization and the argmax algorithm. XLM-R is used as the feature extractor. We compare SUBDP to prior work in the minimal annotation setting (Table 1), where an English dependency treebank is the only annotation that involves human effort. We select target languages from the overlap between those considered by Kurniawan et al. (2021), those covered by XLM-R (Conneau et al., 2020) training corpora, and those supported by CRISS (Tran et al., 2020), resulting in eight languages: Arabic (ar), Hindi (hi), Korean (ko), Turkish (tr), German (de), Spanish (es), French (fr), and Italian (it). We translate English sentences using the unsupervised model CRISS to construct the required bitext. 10 To ensure the quality of the unsupervised bitext, we discard (1) translations where at least 80% of words appear in the corresponding source sentences, which are likely to be copies, (2) those 8 We do not observe further training loss decrease when training for more epochs. The learning rate for SUBDP is tuned to optimize the development loss for German, where the German gold trees remain unused. 9 SUBDP does not provide a set of hard silver trees for LAS and UAS calculation. containing a CRISS language token other than the target language, which are likely to be false translation into another language, and (3) those with 80% or more words appearing in the translated sentence more than once, which are likely to be repetitions. Transferring from an English parser, SUBDP achieves the best UAS across all eight target languages, and the best LAS on six languages out of eight. In addition, we find that SUBDP is consistent across random seeds, with a standard deviation less than 0 . 8 for every number in Table 1. 4.2 Ablation Study We introduce the following baselines with the same annotated data availability for an ablation study: 1. Direct transfer of English models (DT). We train a bi-affine dependency parser on English treebanks, and test the model on other languages. This approach is expected to outperform a random baseline as it has a pretrained cross-lingual language model-based feature extractor, which may implicitly enable cross-lingual transfer. For this baseline, we test both XLM-R and CRISS encoders, as SUBDP benefits from both models. 2. Self-training (ST). Following Kurniawan et al. (2021), we apply an XLM-R DT parser to the target language, 11 and train another parser on the predicted hard trees. 3. Hard projection (Hard). It is intuitive to compare SUBDP against the hard tree projection baseline (Lacroix et al., 2016), where we use the same set of bitext and alignments to project trees to the target languages, keeping only the edges with both sides aligned in a one-to-one alignment. We use the projected trees to train a parser in the target language. 11 We only consider XLM-R as the feature extractor for ST as it achieves better average DT results. 4. Random target parser initialization (RandI). Instead of using the trained English model as the initialization of target parsers, we randomly initialize the weights in this baseline. This approach matches with SUBDP in every component except the target parser initialization. All of the baselines use bi-affine dependency parsers, with pretrained cross-lingual language models (XLM-R or CRISS) as feature extractors. Across all languages, SUBDP significantly outperforms DT with either XLM-R or CRISS word feature extractor. ST does improve over DT consistently, but is much less competitive than SUBDP. This indicates that the gain of SUBDP over prior work is not simply from more powerful word features. While hard treebank projection using the method proposed by Lacroix et al. (2016) is quite competitive, SUBDP consistently produces competitive (Arabic, German, Spanish) or better (Hindi, Korean, Turkish, French, Italian) results. Comparing SUBDP to RandI, we find that initializing the target language parser with a trained source language (English in this work) parser helps improve performance across the board; therefore, source parser initialization should be considered as a general step in future work on zero-shot cross-lingual dependency parsing. Since most existing work has used only one-to-one alignment for annotation projection (Ma and Xia, 2014; Lacroix et al., 2016; Rasooli et al., 2021, inter alia ), we would like to analyze the effect of introducing many-to-one alignment edges in SUBDP. We filter SimAlign BPE argmax to obtain a more conservative version, dropping all many-to-one edges (i.e., those that have a word linked to multiple edges), 12 and compare it to the BPE argmax algorithm (Table 2). While the confident one-to-one alignment achieves further improvement on Arabic and all four nearby languages, we find that the many-to-one BPE argmax alignment is important to the superior transfer performance on Hindi, Korean, and Turkish. Given the fact that the scores are quite similar for Arabic, the results generally suggest using the many-to-one SimAlign BPE argmax alignments for transferring from English to distant languages, while using the more confident one-to-12 This approach is different from Hard as it takes soft source trees as the input, yielding soft target trees as silver labels to train target language parsers. Following Schuster et al. (2019), we use Universal Dependencies v2.0 (McDonald et al., 2013) to evaluate zero-shot cross-lingual transfer from multiple source languages (Table 3). 13 For each language among German (de), Spanish (es), French (fr), Italian (it), Portuguese (pt), and Swedish (sv), annotated treebanks from all other languages and English can be used for training and development purposes. For SUBDP, we generate bitext from all applicable source languages with CRISS. SUBDP outperforms the previous state-of-the-art on German by 13.5 LAS, but under-performs the DT baseline on the other three languages. However, if we start with a trained SUBDP parser for a target language, and use the standard training data (i.e., treebanks in other languages) to further train a bi-13 We do not report performance for Portuguese and Swedish as they are not covered by CRISS; however, the annotated treebanks in these languages are used as source treebanks when applicable. affine dependency parser (DT w/ SUBDP init.), we are able to achieve better results than DT across the board, obtaining competitive or even better LAS than methods that use extra annotations other than source treebanks (Zhang and Barzilay, 2015; Guo et al., 2016). We further evaluate SUBDP in another scenario where a few bitext pairs are available. We consider a larger set of eighteen target languages, including Arabic (ar), Czech (cs), German (de), Spanish (es), Finnish (fi), French (fr), Hindi (hi), Hugarian (hu), Italian (it), Japanese (ja), Korean (ko), Norwegian (no), Portuguese (pt), Russian (ru), Tamil (ta), Telugu (te), Vietnamese (vi), and Chinese (zh). We transfer from English to each target language with Wikimatrix bitext (Schwenk et al., 2021), where the examples are mined with an encoding similarity based bitext miner trained with annotated bitext. We vary the number of Wikimatrix bitext pairs, selecting the number of pairs within the geometric sequence { 50  2 k } 9 k =0 , leaving 10% of the examples for development. On average and for nearby languages (Figure 3), we find that the performance of SUBDP with 50 pairs of bitext is quite close to that with 25K pairs of bitext. Although some distant languages generally require more bitext for further improvement, SUBDP outperforms the direct transfer baseline by a nontrivial margin with a small amount (e.g., 800-1.6K pairs) of bitext. Our work is in line with recent work (Rasooli et al., 2021) which shows that cross-lingual transfer can be done effectively with weak supervision such as Wikipedia links. Our results go further and study the setting of zero additional supervision beyond the source language treebank, demonstrating the potential of zero-shot cross-lingual dependency parsing with zero additional supervision, even between distant languages that do not share vocabulary or subwords. Our work suggests a new protocol for dependency annotation of low-resource languages: (1) train a pretrained multilingual model following existing work such as XLM-R (Conneau et al., 2020) and CRISS (Tran et al., 2020), (2) annotate a small number of bitext pairs or generate bitext with trained unsupervised translation models, and (3) train a zero-shot cross-lingual dependency 25354555657585 avg. Our contribution to zero-shot cross-lingual dependency parsing is arguably orthogonal to contextualized representation alignment (Schuster et al., 2019; Wang et al., 2019), where pretrained multilingual language models are finetuned for better transfer. In contrast, we use the frozen pretrained models to extract features. In addition, projection quality controls by heuristic rulebased filtering (Rasooli and Collins, 2015) may also be combined with SUBDP to further improve the performance. Our results, on the other hand, demonstrate that multilingual pretrained models may have more applications beyond representation-based direct transferinformation extracted from these models without further supervision (e.g., word alignment in this work) may further benefit downstream tasks 6554 (e.g., zero-shot cross-lingual dependency parsing in this work) with appropriate usage. While this work depends on pretrained multilingual models such as CRISS (Tran et al., 2020), which require extensive computational resources to train from scratch, SUBDP may be applied whenever bitext alignment and cross-lingual word embeddings are available. In addition, the required pretrained cross-lingual models are useful for general purposes, and can be applied to other downstream NLP tasks. We suggest that SUBDP can be extended to other scenarios wherever relevant parallel signals are available, such as cross-lingual named entity recognition, cross-lingual constituency parsing, or zero-shot scene graph parsing for images using only the dependency supervision in text. We leave the further exploration of SUBDP on other tasks for future work. We thank Kartik Goyal and Shubham Toshniwal for helpful suggestions on this work. This work is supported in part by a Google Fellowship to FS.
method abstain result abstain abstain abstain abstain other abstain abstain abstain abstain abstain abstain abstain abstain abstain method result abstain other other other other other other other other other other other method objective method other abstain other other method other other abstain method method abstain abstain abstain other abstain abstain abstain method abstain method abstain abstain abstain abstain abstain abstain other abstain method abstain abstain abstain abstain abstain method method abstain method abstain method method method other abstain abstain method abstain abstain method abstain method method abstain method abstain abstain method method method method method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain result abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain result method result result abstain abstain abstain abstain abstain abstain result abstain other abstain abstain result abstain result abstain other abstain abstain abstain abstain method abstain abstain abstain other other