Meta-learning, or learning to learn, is a technique that can help to overcome resource scarcity in cross-lingual NLP problems, by enabling fast adaptation to new tasks. We apply model-agnostic meta-learning ( MAML ) to the task of cross-lingual dependency parsing. We train our model on a diverse set of languages to learn a parameter initialization that can adapt quickly to new languages. We find that meta-learning with pre-training can significantly improve upon the performance of language transfer and standard supervised learning baselines for a variety of unseen, typologically diverse, and low-resource languages, in a few-shot learning setup. The field of natural language processing (NLP) has seen substantial performance improvements due to large-scale language model pre-training (Devlin et al., 2019). Whilst providing an informed starting point for subsequent task-specific fine-tuning, such models still require large annotated training sets for the task at hand (Yogatama et al., 2019). This limits their applicability to a handful of languages for which such resources are available and leads to an imbalance in NLP technology's quality and availability across linguistic communities. Aiming to address this problem, recent research has focused on the development of multilingual sentence encoders, such as multilingual BERT (mBERT) (Devlin et al., 2019) and XLM-R (Conneau et al., 2020), trained on as many as 93 languages. Such pre-trained multilingual encoders enable zero-shot transfer of task-specific models across languages (Wu and Dredze, 2019), offering a possible solution to resource scarcity. Zero-shot transfer, how-(cid:70) Corresponding author: annalangedijk@gmail.com. ever, is most successful among typologically similar, high-resource languages, and less so for languages distant from the training languages and in resource-lean scenarios (Lauscher et al., 2020). This stresses the need to develop techniques for fast cross-lingual model adaptation, that can transfer knowledge across a wide range of typologically diverse languages with limited supervision. In this paper, we focus on the task of universal dependency (UD) parsing and present a novel approach for effective and resource-lean cross-lingual parser adaptation via meta-learning, requiring only a small number of training examples per language (which are easy to obtain even for low-resource languages). Meta-learning is a learning paradigm that leverages previous experience from a set of tasks to solve a new task efficiently. As our goal is fast cross-lingual model adaptation, we focus on optimization-based meta-learning, where the main objective is to find a set of initial parameters from which rapid adaption to a variety of different tasks becomes possible (Hospedales et al., 2020). Optimization-based meta-learning has been successfully applied to a variety of NLP tasks. Notable examples include neural machine translation (Gu et al., 2018), semantic parsing (Huang et al., 2018), pre-training text representations (Lv et al., 2020), word sense disambiguation (Holla et al., 2020) and cross-lingual natural language inference and question answering (Nooralahzadeh et al., 2020). To the best of our knowledge, meta-learning has not yet been explored in the context of dependency parsing. We take inspiration from recent research on universal dependency parsing (Tran and Bisazza, 2019; Kondratyuk and Straka, 2019). We employ an existing UD parsing framework  UDify, a multitask learning model (Kondratyuk and Straka, 2019)  and extend it to perform few-shot model adap-8503 tation to previously unseen languages via metalearning. We pre-train the dependency parser on a high-resource language prior to applying the model-agnostic meta-learning ( MAML ) algorithm (Finn et al., 2017) to a collection of few-shot tasks in a diverse set of languages. We evaluate our model on its ability to perform few-shot adaptation to unseen languages, from as few as 20 examples. Our results demonstrate that our methods outperform language transfer and multilingual joint learning baselines, as well as existing (zero-shot) UD parsing approaches, on a range of language families, with the most notable improvements among the low-resource languages. We also investigate the role of a pre-training language as a starting point for cross-lingual adaptation and the effect of typological properties on the learning process. In meta-learning, the datasets are separated into episodes that correspond to training tasks. Each episode contains a support and a query set, that include samples for adaptation and evaluation, respectively. Meta-learning serves as an umbrella term for algorithms from three categories: Metric-based methods classify new samples based on their similarity to the support set (e.g. Snell et al., 2017). Model-based methods explicitly store meta-knowledge within their architectures  e.g. through an external memory (Santoro et al., 2016). Optimization-based methods, on which we focus, estimate parameter initializations that can be fine-tuned with a few steps of gradient descent (e.g. Finn et al., 2017; Nichol and Schulman, 2018). Finn et al. (2017) proposed MAML to learn parameter initializations that generalize well to similar tasks. During the meta-training phase, MAML iteratively selects a batch of episodes, on which it fine-tunes the original parameters given the support set in an inner learning loop , and tests it on the query set. The gradients of the query set with respect to the original parameters are used to update those in the outer learning loop , such that these weights become a better parameter initialization over iterations. Afterwards, during meta-testing , one selects a support set for the test task, adapts the model using that set and evaluates it on new samples from the test task. MAML has provided performance benefits for cross-lingual transfer for tasks such as machine translation (Gu et al., 2018), named entity recognition (Wu et al., 2020), hypernymy detection (Yu et al., 2020) and mapping lemmas to inflected forms (Kann et al., 2020). The closest approach to ours is by Nooralahzadeh et al. (2020), who focus on natural language inference and question answering. Their method, X-MAML , involves pre-training a model on a high-resource language prior to applying MAML . This yielded performance benefits over standard supervised learning for cross-lingual transfer in a zero-shot and fine-tuning setup (albeit using 2500 training samples to fine-tune on test lan-guages). The performance gains were the largest for languages sharing morphosyntactic features. Besides the focus on dependency parsing, our approach can be distinguished from Nooralahzadeh et al. (2020) in several ways. We focus on fast adaptation from a small number of examples (using only 20 to 80 sentences). Whilst they use one language for meta-training, we use seven languages, with the aim of explicitly learning to adapt to a variety of languages. The Universal Dependencies project is an ongoing community effort to construct a cross-linguistically consistent morphosyntactic annotation scheme (Nivre et al., 2018). The project makes results comparable across languages and eases the evaluation of cross-lingual (structure) learning. The task of dependency parsing involves predicting a dependency tree for an input sentence, which is a directed graph of binary, asymmetrical arcs between words. These arcs are labeled and denote dependency relation types, which hold between a head -word and its dependent . A parser is tasked to assign rankings to the space of all possible dependency graphs and to select the optimal candidate. Dependency parsing of under-resourced languages has since long been of substantial interest in NLP. Well-performing UD parsers, such as the winning model in the CoNLL 2018 Shared Task by Che et al. (2018), do not necessarily perform well on low-resource languages (Zeman et al., 2018). Cross-lingual UD parsing is typically accomplished by projecting annotations between languages with parallel corpora (Agic et al., 2014), through model transfer (e.g. Guo et al., 2015; Ammar et al., 2016; Ahmad et al., 2019), through hybrid methods combining annotation projections and model transfer (Tiedemann et al., 2014), or by aligning word embeddings across languages (Schuster et al., 2019). State-of-the-art methods for cross-lingual dependency parsing exploit pre-trained mBERT with a dependency parsing classification layer that is fine-tuned on treebanks of high-resource languages, and transferred to new languages: Wu and Dredze (2019) only fine-tune on English, whereas Tran and Bisazza (2019) experiment with multiple sets of fine-tuning languages. Including diverse language families and scripts benefits transfer to low-resource languages, in particular. UDify, the model of Kondratyuk and Straka (2019), is jointly fine-tuned on data from 75 languages, with a multi-task learning objective that combines dependency parsing with predicting part-of-speech tags, morphological features, and lemmas. stn et al. (2020), instead, freeze the mBERT parameters and train adapter modules that are interleaved with mBERT's layers, and take a language embedding as input. This embedding is predicted from typological features. Model performance strongly relies on the availability of those features, since using proxy embeddings from different languages strongly degrades low-resource languages' performance. We use data from the Universal Dependencies v2.3 corpus (Nivre et al., 2018). We use treebanks from 26 languages that are selected for their typological diversity. We adopt the categorization of high-resource and low-resource languages from Tran and Bisazza (2019) and employ their set of training and test languages for comparability. The set covers languages from six language families (Indo-European, Korean, Afro-Asiatic, Uralic, Dravidian, Austro-Asiatic). Their training set ( expMix ) includes eight languages: English, Arabic, Czech, Hindi, Italian, Korean, Norwegian, and Russian. These languages fall into the language families of Indo-European, Korean and Afro-Asiatic and have diverse word orders (i.e. VSO, SVO and SOV). Joint learning on data from this diverse set yielded state-of-the-art zero-shot transfer performance on low-resource languages in the experiments of Tran and Bisazza (2019). Per training language we use up to 20,000 example trees, predicting dependency arc labels from 132 classes total. We select Bulgarian (Indo-European) and Telugu (Dravidian) as validation languages to improve generalization to multiple language families. The 16 test languages cover three new language families that were unseen during training, i.e. Austro-Asiatic, Dravidian, and Uralic. Furthermore, three of our test languages (Buryat, Faroese, and Upper Sorbian) are not included in the pre-training of mBERT. We refer the reader to Appendix B for details about the treebank sizes and language families. The UDify model concurrently predicts part-of-speech tags, morphological features, lemmas and dependency trees (Kondratyuk and Straka, 2019). UDify exploits the pre-trained mBERT model (De-vlin et al., 2019), that is a self-attention network with 12 transformer encoder layers. The model takes single sentences as input. Each sentence is tokenized into subword units using mBERT's word piece tokenizer, after which contextual embedding lookup provides input for the self-attention layers. A weighted sum of the outputs of all layers is computed (Equation 1) and fed to a task-specific classifier. Here, e t denotes the contextual output embeddings for task t . In our case, t indicates UD-parsing. In contrast to the multi-task objective of the original UDify model, our experiments only involve UD-parsing. The term B ij represents the mBERT representation for layer i = 1 , ..., 12 at token position j . The terms  and  denote trainable scalars, where the former applies to mBERT and the latter scales the normalized averages. For words that were tokenized into multiple word pieces, only the first word piece was fed to the UD-parsing classifier. The UD-parsing classifier is a graph-based biaffine attention classifier (Dozat and Manning, 2017) that projects the embeddings e tj through arc-head and arc-dep feedforward layers. The resulting outputs are combined using biaffine attention to produce a probability distribution of arc heads for each word. Finally, the dependency tree is decoded using the Chu-Liu/Edmonds algorithm (Chu, 1965; Edmonds, 1967). We refer the reader to the work of Kondratyuk and Straka (2019) for further details on the architecture and its training procedure. We apply first-order 1 MAML to the UDify model. The model's self-attention layers are initialized with parameters from mBERT and the classifier's feedforward layers are randomly initialized. The model is pre-trained on a high-resource language using standard supervised learning and further meta-trained on a set of seven languages with MAML . It is then evaluated using meta-testing. We refer to MAML with pre-training as simply MAML . The meta-learning procedure is visualized in Figure 1 and can be described as follows: Step 1 Pre-train on a high-resource language to yield the initial parameters  . Step 2 Meta-train on all other training languages. For each language i , we partition the UD training data into two disjoint sets, D train i and D test i , and perform the following inner loop: 1. Temporarily update the model parameters  i with stochastic gradient descent on support set S , sampled from D train i , with learning rate  for k gradient descent adaptation steps. When using a single gradient step, the update becomes:  i       L ( i ) (2) 2. Compute the losses of the model parameters  i using the query set Q , sampled from D test i , denoted by L i ( i ) . 1 For more details on first-order versus second-order, see Finn et al. (2017); Holla et al. (2020). Step 3 Sum up the test losses and perform a meta-update in the outer learning loop on the model with parameters  using the learning rate  :      (cid:88) i   L i ( i ) (3) In our experiments, the update is a first-order approximation, replacing   L i ( i ) by   i L i ( i ) . Step 4 After meta-training, we apply meta-testing to unseen languages. For each language, we sample a support set S from the UD training data. We then fine-tune our model on S , and evaluate the model on the entire test set. Thereby, meta-testing mimics the adaptation from the inner loop. We repeat this process multiple times to get a reliable estimate of how well the model adapts to unseen languages. We extend the existing UDify code 2 to be used in a meta-learning setup. All of our code is publicly available. 3 5.1 Training and evaluation Pre-training In the main body of the paper, we consider the pre-training languages English and Hindi to measure the impact of pre-training prior to cross-lingual adaptation, and to draw more general conclusions about how well MAML generalizes with typologically different pre-training languages. English and Hindi differ in word order (SVO versus SOV), and Hindi treebanks have a larger percentage of non-projective dependency trees (Mannem et al., 2009), where dependency arcs are allowed to cross one another. Non-projective trees are more challenging to parse (Nivre, 2009). Pre-training on Hindi allows us to test the effects of projectivity on cross-lingual adaptation. To ensure that our findings are not specific to the pre-training languages of English and Hindi, Appendix D reproduces a subset of experiments for the pre-training languages Italian and Czech, reporting results for monolingual baselines, a non-episodic baseline, and MAML . Italian and Czech are high-resource languages as well, but are from two different subfamilies of the family of Indo-European languages and also differ in their percentage of non-projective dependency trees. 2 github.com/Hyperparticle/udify 3 github.com/annaproxy/udify-metalearning 8506 Meta-training We apply meta-training using seven languages listed in Section 3, excluding the pre-training language from meta-training. We train for 500 episodes per language, using a cosine-based learning rate scheduler with 10% warm-up. We use the Adam optimizer (Kingma and Ba, 2015) in the outer loop and SGD in the inner loop (Finn et al., 2017). Support and query sets are of size 20. Due to the sequence labelling paradigm, the number of shots per class varies per batch. When | S | = 20 , the average class will appear 16 times. To select hyperparameters, we independently vary the amount of updates k and the learning rates in the inner loop and outer loop for mBERT and the parser, while performing meta-validation with the languages Bulgarian and Telugu. To meta-validate, we follow the procedure described in Section 4.2 for both languages, mimicking the meta-testing setup with a support set size of 20. The hyperparameters are estimated independently for Hindi and English pre-training (see Appendix A). Meta-testing At meta-testing time, we use SGD with the same learning rates and the same k used in the inner loop during meta-training. We vary the support set size | S |  { 20 , 40 , 80 } . We define several baselines, that are evaluated using meta-testing, i.e. by fine-tuning the models on a support set of a test language prior to evaluation on that language. This allows us to directly compare their ability to adapt quickly to new languages with that of the meta-learner. Monolingual baselines ( EN , HIN ) These baselines measure the impact of meta-training on data from seven additional languages. The model is initialized using mBERT and trained using data from English ( EN ) or Hindi ( HIN ), without meta-training. Multilingual non-episodic baseline ( NE ) Instead of episodic training, this baseline treats support and query sets as regular mini-batches and updates the model parameters directly using a joint learning objective, similar to Kondratyuk and Straka (2019) and Tran and Bisazza (2019). The model is pre-trained on English or Hindi and thus indicates the advantages of MAML over standard supervised learning. The training learning rate and meta-testing learning rate are estimated separately, since there is no inner loop update in this setup. MAML without pre-training We evaluate the effects of pre-training by running a MAML setup without any pre-training. Instead, the pre-training language is included during meta-training as one of now eight languages. MAML without pre-training is trained on 2000 episodes per language. Meta-testing only The simplest baseline is a decoder randomly initialized on top of mBERT, without pre-training and meta-training. Dependency parsing is only introduced at meta-testing time. Hyperparameter selection and evaluation is performed using Labeled Attachment Score (LAS) as computed by the CoNLL 2018 Shared Task evaluation script. 4 LAS evaluates the correctness of both the dependency class and dependency head. We use the standard splits of Universal Dependencies for training and evaluation when available. Otherwise, we remove the meta-testing support set from the test set prior to evaluation. We train each model with seven different seeds and compare MAML to a monolingual baseline and NE using paired t -tests, adjusting for multiple comparisons using Bonfer-roni correction. MAML with English pre-training We report the mean LAS for models pre-trained on English in Table 1. We compare these results to related approaches that use mBERT and have multiple training languages. With support set size 20, MAML already outperforms the zero-shot transfer setup of Tran and Bisazza (2019) for all test languages, except Persian and Urdu. MAML is competitive with UDify (Kondratyuk and Straka, 2019) and UDapter (stn et al., 2020) for low-resource languages, despite the stark difference in the number of training languages compared to UDify 5 (75), and without relying on fine-grained typological features of languages, as is the case for UDapter. MAML consistently outperforms the EN and NE baselines. Large improvements over the EN baseline are seen on low-resource and non-Germanic languages. The difference between MAML and the baselines increases with | S | . The largest improvements over NE are on Tamil and Japanese, 4 universaldependencies.org/conll18/evaluation.html 5 UDify is trained on the low-resource languages, while we only test on them. For a fair comparison, we only list UDify results on languages with a small amount of sentences ( < 80) in the training set, to mimic a few-shot generalisation setup. however NE outperforms MAML on Hungarian and Urdu. MAML consistently outperforms NE on low-resource languages, with an average 1.1% improvement per low-resource language for | S | = 20 , up to a 2.2% average improvement for | S | = 80 . MAML with Hindi pre-training The results for models pre-trained on Hindi can be seen in Table 3. Although there are large differences between the monolingual EN and HIN baselines, both MAML ( HIN ) and NE ( HIN ) achieve, on average, similar LAS scores to their English counterparts. MAML still outperforms NE for the majority of languages: the mean improvement on low-resource languages is 0.8% per language for | S | = 20 , which increases to 1.6% per language for | S | = 80 . Other pre-training languages The full results for the two other pre-training languages, Italian and Czech, are listed in Appendix D. Here, too, MAML outperforms its NE counterpart. The NE baseline is stronger for more languages than in our main experiments. For | S | = 20 , the mean improvements per unseen language are 0.91% and 0.47% when pre-training on Italian and Czech, respectively. For | S | = 80 , the improvements are 2.18% and 1.75%. MAML without (pre-)training We investigate the effectiveness of pre-training by omitting the pre-training phase. A comparison between MAML and MAML without pre-training is shown in Table 2. MAML without pre-training underperforms for most languages and its performance does not increase as much with a larger support set size. This suggests that pre-training provides a better starting point for meta-learning than plain mBERT. When meta-testing only  i.e. omitting both pretraining and meta-training  the fine-tuned model reaches a mean LAS of 6.9% over all test languages for | S | = 20 , increasing to 15% for | S | = 80 , indicating that meta-testing alone is not sufficient 8508 | S | = 20 | S | = 40 | S | = 80 Language HIN NE MAML HIN NE MAML HIN NE MAML Low-Resource Languages Armenian 48.41 63.30 63.76 48.87 63.41 64.17 49.70 63.59 64.76 Breton 34.06 62.09 61.56 36.09 62.40 62.47 38.95 63.05 63.75 Buryat  24.24 25.05 26.27 24.71 25.18 26.79 25.54 25.40 27.37 Faroese  50.72 65.31 66.82 52.30 65.57 67.31 54.64 66.17 68.25 Kazakh 49.80 53.77 54.23 49.90 53.94 54.45 50.49 54.08 55.00 U.Sorbian  36.22 53.36 54.97 37.08 53.58 55.64 38.22 53.94 56.56 Mean 40.57 53.81 54.60 41.49 54.01 55.14 42.92 54.37 55.95 High-Resource Languages Finnish 50.49 64.05 64.64 50.93 64.20 65.05 51.79 64.40 65.61 French 31.16 64.44 65.73 31.59 64.44 65.68 33.39 64.42 65.69 German 44.83 74.40 75.15 45.46 74.41 75.23 46.65 74.46 75.31 Hungarian 46.72 60.98 62.51 46.97 61.33 62.89 47.91 61.68 62.91 Japanese 40.25 39.97 41.96 43.03 40.56 43.61 46.87 41.58 45.90 Persian 28.60 53.73 53.63 29.51 53.85 54.00 31.11 54.06 54.53 Swedish 46.96 79.24 79.89 47.73 79.32 80.14 49.15 79.31 80.21 Tamil 46.51 39.44 39.57 47.35 39.84 40.84 48.55 40.73 42.81 Urdu 67.72 50.64 49.16 67.96 50.93 50.16 68.17 51.50 51.57 Vietnamese 26.96 42.13 42.12 27.92 42.23 42.37 29.61 42.46 42.87 Mean 43.02 56.9 57.44 43.85 57.11 58.0 45.32 57.46 58.74 Mean 42.1 55.74 56.37 42.96 55.95 56.92 44.42 56.3 57.69 Table 3: Mean LAS aligned accuracy per unseen language, for models pre-trained on Hindi. Further Analysis Performance increases over the monolingual baselines vary strongly per language  e.g. consider the difference between Japanese and French in Table 1. The performance increase is largest for languages that differ from the pre-training language with respect to their syntactic properties. We conduct two types of analysis, based on typological features and projectivity, to quantify this effect and correlate these properties to the performance increase over monolingual baselines. 7 Firstly, we use 103 binary syntactic features from URIEL (Littell et al., 2017) to compute the syntactic cosine similarities (denoted  ) between languages. With this metric, a language such as Italian is syntactically closer to English (  = 0 . 86 ) than Urdu (  = 0 . 62 ), even though they are both Indo-European. For each unseen language, we collect the cosine similarities to each (pre-)training language. Then, we collect the difference in performance between the monolingual baselines and the NE or MAML setups for | S | = 20 . For each training language, we compute the correlations between performance increases for the test languages and their similarity to this training language, visualised 6 Full results can be found in Appendix C. 7 No clear correlation was found by Tran and Bisazza (2019). in Figure 2. When pre-training on Hindi, there is a significant positive correlation with syntactic similarity to English and related languages. When pre-training on English, a positive correlation is seen with similarity to Hindi and Korean. Positive correlations imply that on unseen languages, improvement increases when similarity to the training language increases. Negative correlations mean there is less improvement when similarity to the training languages increases, suggesting that those languages do not contribute as much to adaptation. On average, the selection of meta-training languages contributes significantly to the increase in performance for the Hindi pre-training models. This effect is stronger for MAML (HIN) ( p = 0 . 006 ) than NE (HIN) ( p = 0 . 026 ), which may indicate that the meta-training procedure is better at incorporating knowledge from those unrelated languages. Secondly, we analyze which syntactic features impact performance most. We correlate individual URIEL features with MAML 's performance increases over monolingual baselines (see Figure 3). Features related to word order and negation show a significant correlation. Considering the presence of these features in both pre-training languages of MAML , a pattern emerges: when a feature is absent in the pre-training language, there is a positive correlation with increase in performance. Similarly, when a feature is present in the pre-training lan-8509 NE ( EN ) NE ( HIN ) MAML ( EN ) MAML ( HIN ) Arabic Czech Italian Norwegian Russian Hindi Korean English Mean L a n g u a g e f o r s i m il a r i t y -0.31 0.62* -0.23 0.66* -0.26 0.54* -0.19 0.62* -0.5* 0.71* -0.41 0.77* -0.54* 0.63* -0.48 0.69* -0.48 0.6* -0.42 0.67* 0.61* -0.58* 0.59* -0.53* 0.67* -0.81* 0.63* -0.75* -0.6* 0.73* -0.55* 0.78* -0.36 0.55* -0.29 0.66* 1.0 0.5 0.0 0.5 1.0 Figure 2: Spearman's  between the performance increase over the monolingual baseline and the cosine similarity to the syntax of training languages ( y -axis) for models using pre-training ( x -axis). This indicates that MAML is successfully adapting to these specific features during meta-training. We analyzed MAML 's performance improvements over NE on each of the 132 dependency relations, and found that they are consistent across relations. 8 Lastly, we detect non-projective dependency trees in all datasets. The Hindi treebank used has 14% of non-projective trees, whereas English only has 5%. 9 We correlate the increase in performance with the percentage of non-projective trees in a language's treebank. The correlation is significant for NE (EN) (  = 0 . 46 , p = 0 . 01 ) and MAML (EN) (  = 0 . 42 , p = 0 . 03 ). Figure 4 visualizes the correlation for MAML (EN). We do not find significant correlations for models pre-trained on Hindi. This suggests that a model trained on a mostly projective language can benefit more from further training on non-projective languages than the other way around. The same trend is observed when comparing models pre-trained on Italian and Czech, that also differ in the percentage of nonprojective trees (Appendix D). Our experiments confirm that meta-learning, specifically MAML , is able to adapt to unseen languages on the task of cross-lingual dependency parsing 8 The same holds for the 37 coarse-grained UD relations. 9 Full results can be found in Appendix B. MAML (EN)MAML (HIN) SVO ( ) Adposition before noun ( ) Possessor after noun ( ) Negative word after verb ( ) Negative word before object ( ) Prosubject word ( ) VOX ( ) Oblique after verb ( ) SOV ( ) Adposition after noun ( ) Object headmark ( ) Any redup. more effectively than a non-episodic model. The difference between both methods is most apparent for languages that differ strongly from those in the training set (e.g. Japanese in Table 1) where effective few-shot adaptation is crucial. This shows that MAML is successful at learning to learn from a few examples, and can efficiently incorporate new information. Furthermore, we see a clear increase in performance for MAML when increasing the test support set size, while NE only slightly improves. This suggests that MAML may be a promising method for cross-lingual adaptation more generally, also outside of the few-shot learning scenario. Our ablation experiments on pre-training show that it is beneficial for MAML to start from a strong set of parameters, pre-trained on a high resource 8510 language. Thereby, the pre-training is not dependent on a specific language. MAML performs well with a variety of pre-training languages, although improvements for unseen languages vary. When a model is pre-trained on English, there is a large positive correlation for improvements in languages that are syntactically dissimilar to English, such as Japanese and Tamil. During meta-training, dissimilar training languages such as Hindi most contribute to the model's ability to generalize. Syntactic features, especially those related to word order, which have already been learned during pretraining, require less adaptation. The same is true, vice versa, for Hindi pre-training. This effect is also observed, though only in one direction, when correlating performance increase with non-projectivity. It is beneficial to meta-train on a set of languages that vary in projectivity after pre-training on one which is mostly projective. However, not all variance is explained by the difference in typological features. The fact that MAML outperforms MAML without pretraining suggests that pre-training also contributes language-agnostic syntactic features, which is indeed the overall goal of multi-lingual UD models. In this paper, we present a meta-learning approach for the task of cross-lingual dependency parsing. Our experiments show that meta-learning can improve few-shot universal dependency parsing performance on unseen, unrelated test languages, including low-resource languages and those not covered by mBERT. In addition, we see that it is beneficial to pre-train before meta-training, as in the X-MAML approach (Nooralahzadeh et al., 2020). In particular, the pre-training language can affect how much adaptation is necessary on languages that are typologically different from it. Therefore, an important direction for future research is to investigate a wider range of pre-training/meta-training language combinations, based on specific hypotheses about language relationships and relevant syntactic features. Task performance may be further improved by including a larger set of syntax-related tasks, such as POS-tagging, to sample from during meta-training (Kondratyuk and Straka, 2019).
abstain method objective result abstain abstain abstain abstain abstain abstain abstain abstain objective abstain abstain abstain abstain objective method abstain method method objective objective other other other other method other other other other other other other other other abstain method objective other other other other other other other other other other other other other other method method method abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method method abstain abstain abstain abstain abstain abstain method method abstain abstain abstain method abstain abstain method abstain abstain abstain abstain method method method method abstain method method method method abstain abstain method method abstain method method method abstain abstain abstain method method abstain method method method method abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain method method method abstain result abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain other abstain abstain method method abstain method method abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain other abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain result abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method result result abstain abstain abstain