Massively Multilingual Transformer based Language Models have been observed to be surprisingly effective on zero-shot transfer across languages, though the performance varies from language to language depending on the pivot language(s) used for fine-tuning. In this work, we build upon some of the existing techniques for predicting the zero-shot performance on a task, by modeling it as a multi-task learning problem. We jointly train predictive models for different tasks which helps us build more accurate predictors for tasks where we have test data in very few languages to measure the actual performance of the model. Our approach also lends us the ability to perform a much more robust feature selection, and identify a common set of features that influence zero-shot performance across a variety of tasks. Multilingual models like mBERT (Devlin et al., 2019) and XLM-R (Conneau et al., 2020) have been recently shown to be surprisingly effective for zero-shot transfer (Pires et al., 2019) (Wu and Dredze, 2019), where on fine-tuning for a task on one or a few languages, called pivots , they can perform well on languages unseen during training. The zero-shot performance however, is often not uniform across the languages and the multilingual models turn out to be much less effective for low resource languages (Wu and Dredze, 2020; Lauscher et al., 2020) and the languages that are typologically distant from the pivots (Lauscher et al., 2020). What affects the zero-shot transfer across different languages is a subject of considerable interest and importance (K et al., 2020; Pires et al., 2019; Wu and Dredze, 2019; Lauscher et al., 2020), however there is little conclusive evidence and a few papers even show contradictory findings. Lauscher et al. (2020) recently, showed that it is possible to predict the zero shot performance of  Equal contribution mBERT and XLM-R on different languages by formulating it as a regression problem, with pretraining data size and typological similarities between the pivot and target languages as the input features, and the performance on downstream task as the prediction target. Along similar lines Srinivasan et al. (2021) and Dolicki and Spanakis (2021) explore zero-shot performance prediction with a larger set of features and different regression techniques. However, the efficacy of these solutions are severely limited by the lack of training data, that is, the number of languages for which performance metrics are available for a given task. For instance, for most tasks in the popular XTREME-R (Ruder et al., 2021) benchmark, there are data points for 7-11 languages. This not only makes zero-shot performance prediction a challenging problem, but also a very important one because for practical deployment of such multilingual models, one would ideally like to know its performance for all the languages the model is supposed to handle. As Srinivasan et al. (2021) shows, accurate performance predictors can also help us build better and fairer multilingual models by suggesting data labeling strategies. In this work, we propose multi-task learning (Zhang and Yang, 2017) as an approach to mitigate training-data constraints and consequent over-fitting of the performance predictors to tasks and/or datasets. The contributions of our work are fourfold. First, we experiment with different multitask learning approaches, such as Group Lasso (Yuan and Lin, 2006), Collective Matrix Factorization (Cortes, 2018), Multi-Task Deep Gaussian Process Regression (Bonilla et al., 2008) and Meta Agnostic Meta Learning (Finn et al., 2017) for 11 tasks. We observe an overall 10% reduction in performance prediction errors compared to the best performing single-task models. The gains are even stronger when we just consider the tasks with very few data points (  10 ), where we see a 20% 5454 drop in the mean absolute errors. Second, an interesting consequence of modelling this problem via multi-task learning is that we are able to predict performance on low resource languages much more accurately, where in some cases single-task approaches may perform even worse than the simple averaging baselines. Third, apart from the features used for zero-shot performance prediction in the previous work (Lauscher et al., 2020; Srinivasan et al., 2021; Dolicki and Spanakis, 2021), we also utilize metrics quantifying the quality of multilingual tokenizers as proposed in (Rust et al., 2021) as features in our predictive models, which turn out to have strong predictive power for certain tasks. To the best of our knowledge, our work is the first to explore the impact of tokenizer quality specifically on zero-shot transfer. And fourth, our multi-task framework in general lends us with a much more robust selection of features affecting the zero-shot performance. This, in turn, lets us investigate the critical open question on what influences the zero-shot performances across languages more rigorously. As we shall see, our findings corroborate some of the previous conclusions, while others are extended or annulled. Zero Shot Transfer. Multilingual models like mBERT (Devlin et al., 2019) and XLM-R (Con-neau et al., 2020) have shown surprising effectiveness in zero-shot transfer, where fine-tuning the MMLM on a task in some source language often leads to impressive performance on the same task in other languages as well without explicitly training on them. Pires et al. (2019) first observed this phenomenon for NER (Tjong Kim Sang, 2002; Tjong Kim Sang and De Meulder, 2003; Levow, 2006) and POS tagging (Nivre et al., 2018) tasks. Concurrently, Wu and Dredze (2019) also showed this surprisingly cross lingual transfer ability of mBERT additionally on tasks like Document Classification (Schwenk and Li, 2018), Natural Language Inference (Conneau et al., 2018) and Dependency Parsing (Nivre et al., 2018). Factors Affecting Zero Shot Transfer. Pires et al. (2019) showed that vocabulary memorization played little role in zero-shot generalization as language pairs with little word piece overlap also exhibited impressive crosslingual performance. K et al. arrived at a similar conclusion by training BERT on an artificially generated language to zero out the word overlap with the target languages, and observed only minor drops in the performance compared to training the model on English. On the contrary Wu and Dredze (2019), observed strong correlations between the sub-word overlap and the zero-shot performance in four out of five tasks. Wu and Dredze (2020) showed that mBERT performed much worse for zero-shot transfer to low resource languages (i.e., less pre-training data) than high resource ones on POS Tagging, NER and Dependency Parsing tasks. Lauscher et al. (2020) also had a similar observation on tasks like XNLI and XQuAD (Artetxe et al., 2020), though they found that the zero-shot performance on NER, POS tagging and Dependency Parsing tasks might not strictly depend on the pre-training size and could be better explained by different linguistic relatedness features like syntactic and phonological similarities between the language pair. Similar dependence on the typological relatedness such as word order had also been observed by Pires et al. (2019). Performance Prediction. Prior work has explored predicting the performance of machine learning models from unlabelled data by either measuring (dis)agreements between multiple classifiers (Pla-tanios et al., 2014, 2017) or by utilizing underlying information about data distribution (Domhan et al., 2015). In the context of NLP Birch et al. (2008) explored predicting the performance of a Machine Translation system by utilizing different explanatory variables for the language pairs. Lin et al. (2019) proposed a learning to rank approach to choose transfer languages for cross lingual learning using several linguistic and dataset specific features. Recently, there has been an interest in predicting the performance of NLP models without actually training or testing them, by formulating it as a regression problem. Xia et al. (2020) showed that using experimental settings for an NLP experiment as inputs it is possible to accurately predict the performance on different languages and model architectures.Ye et al. (2021) extended this work by proposing methods to do a fine-grained estimation of the performance as well as predicting well-callibrated confidence intervals. Specifically predicting the zero-shot performance of MMLMs was first explored in Lauscher et al. (2020), where they used a linear regression model to estimate the cross-lingual transfer performance based on pretraining data size and linguistic relatedness features. Srinivasan et al. (2021) tackled this problem by utilizing XGBoost Regressor for the prediction along with a larger set of features. Dolicki and Spanakis (2021) explored individual syntactic features for zero-shot performance prediction instead of working with aggregate similarity values, and showed about 2 to 4 times gain in performance. We extend all of these works by considering a multi-task learning approach, where performance prediction in a task utilizes not only the data available for that task, but also the patterns observed for other tasks. We begin by defining the multi-task performance prediction problem and then describe the different linguistic and MMLM specific features used. Consider a pre-trained multilingual model M , trained using self supervision on a set of languages L . Let T be the set of downstream NLP tasks, P be the set of pivot (source) languages for which training data is available for the downstream tasks for fine-tuning and T be the set of target languages for which validation/test data is available. Note that P  L and T  L . We use the zero-shot setting similar to Lauscher et al. (2020) which enforces P and T to be disjoint sets 1 , i.e., P  T =  . We then define y M , t p,t  R as the zero-shot performance on language t  T on finetuning M on task t  T in pivot language p  P . Let x M p,t  R n be the n -dimensional feature vector representing the corresponding train-test configuration. Since for our experiments we train and evaluate the performance prediction for a single model at a time, we will simplify the notations to y t p,t and x p,t . The predictor model can then be defined as the function f  ,  : R n  T  R , where   R d g denotes the shared parameters across the tasks and the task specific parameters are given by   R d s | T | . The objective function for training such a predictor model can be defined as: J ( , ) = (cid:88) t  T (cid:88) p P (cid:88) t T  f ( x p,t , t ;  , )  y t p,t  22 +  g    1 +  s    1 , 1 +  group    1 ,q (1) 1 Though beyond the scope of the current work, it is possible to extend this to a few-shot setting as discussed in Srinivasan et al. (2021). The second and third terms regularize the global and task specific parameters independently, while the last term, l 1 /l q norm with q > 1 , ensures a block sparse selection of the task specific parameters. This term ensures a multi-task learning behavior even when there are no parameters shared across the tasks (i.e.,  =  ) through selection of common features across the tasks. Setting  =  and  group = 0 leads to the single task setup of Lauscher et al. (2020) and Srinivasan et al. (2021). We divide the set of features into two higher level categories, viz. the pairwise features defined for the pivot and target that measure the typological relatedness of the languages, and the individual features defined for the target language reflecting the state of its representation in M . Instead of directly using the different typological properties of the the two languages as features, we use the pairwise relatedness to avoid feature explosion. Subword Overlap : We define the subword overlap as the percentage of unique tokens that are common to the vocabularies of both the pivot and target languages. Let V p and V t be the subword vocabularies of p and t . The subword overlap is then defined as : o sw ( p, t ) = | V p  V t | | V p  V t | (2) Similarity between Lang2Vec vectors : Following Lin et al. (2019) and Lauscher et al. (2020), we compute the typological relatedness between p and t from the linguistic features provided by the URIEL project (Littell et al., 2017). We use syntactic ( s syn ( p, t ) ), phonological similarity ( s pho ( p, t ) ), genetic similarity ( s gen ( p, t ) ) and geographic distance ( d geo ( p, t ) ). For details, please see Littell et al. (2017). Pre-training Size : We use the log 10 of the size (in words) of the pre-training corpus in the target language, SIZE ( t ) , as a feature. Rare Typological Traits : Srinivasan et al. (2021) proposed this metric to capture the rarity of the typological features of a language in the representation of M . Every typological feature in WALS 5456 database is ranked based on the amount of pretraining data for the languages that contain the feature. For the language t , Mean Reciprocal Rank (MRR) of all of its features is then calculated and used as a feature  WMRR ( t ) . Tokenizer Features : In their recent work, Rust et al. (2021) proposed two metrics, viz. tokenizer's fertility and proportion of continued words, to evaluate the quality of multilingual tokenizers on a given language. For target t , they define the tokenizer's fertility, FERT ( t ) , as the average number of sub-words produced for every tokenized word in t 's corpus. On the other hand, the proportion of continued words, PCW ( t ) , measures how often the tokenizer chooses to continue a word across at least two tokens. They show that the multilingual models perform much worse on a task than their monolingual counterparts when the values of these metrics are higher for the multilingual tokenizer. We include FERT ( t ) and PCW ( t ) as features. An important thing to note here is that the we do not use identity of a language as a feature while training the models, hence the performance prediction models are capable of generating predictions on new languages unseen during training. However, if the features of the new languages deviate significantly from the features seen during training, the predictions are expected to be less accurate as also observed in Xia et al. (2020); Srinivasan et al. (2021) and is one of the main reasons for exploring a multi-task approach. We extensively experiment with a wide-array of multi-task as well as single-task regression models to provide a fair comparison between different approaches to zero-shot performance prediction. Average Score Within a Task (AWT) : The performance for a pivot-target pair ( p , t ) on a task t is approximated by taking the average of the performance on all other target languages (pivot being fixed) in the same task t , i.e., f ( x p,t , t ) = 1 |T | 1 (cid:80) t  T { t } y t p,t  . Average Score across the Tasks (AAT) : Here instead of averaging over all the target languages within a task, we approximate the performance on a given target language by averaging the scores for that language across the other tasks, i.e., f ( x p,t , t ) = 1 | T | 1 (cid:80) t   T { t } y t  p,t . Lasso Regression : Lauscher et al. (2020) train different linear regression models for each task. Along similar lines, we experiment with linear regression, but also add an L1 regularization term, as we observed it usually leads to better predictors. XGBoost Regressor : As shown in Srinivasan et al. (2021), XGBoost (Chen and Guestrin, 2016) generally obtains impressive performance on this task, and hence we include it in our experiments as well. Group Lasso : l 1 /l q norm based block-regularization has been shown to be effective for multi-task learning in the setting of multi-linear regression (Yuan and Lin, 2006; Argyriou et al., 2008). For each task, consider separate linear regression models represented by the weight matrix   R n | T | . The l 1 /l q regularization term is given as:    1 ,q = (cid:80) nj =1 ( (cid:80) | T | t =1 |  j t | q ) 1 /q , where  j t denotes the weight for the feature j in the task t . For q > 1 , minimizing this term pushes the l q -norms corresponding to the weights of a given feature across the tasks to be sparse, which encourages multiple predictors to share similar sparsity patterns. In other words, a common set of features is selected for all the tasks. We use q = 2 for the group regularization term. Since this can be restrictive in certain scenarios, some natural extensions to Group Lasso, such as Dirty Models (Jalali et al., 2010) and Multi Level Lasso (Lozano and Swirszcz, 2012), have been proposed that separate out the task specific and global parameters. We experimented with these methods and observed equivalent or worse performance compared to Group Lasso. Collective Matrix Factorization (CMF) with Side Information : Low rank approximation for the task weights matrices forms one family of methods for multi-task learning (Zhang and Yang, 2017; Pong et al., 2010; Ando et al., 2005). As a direct analogue with collaborative filtering, here we can think of the tasks as users and pivot-target pairs as items . Consider the matrix Y  R | T ||PT | , where each element of the matrix correspond to y t p,t . We can then decompose the matrix into task and language-pair specific factors as Y  TLT (3) 5457 where T  R | T | d latent and L  R |PT | d latent are the task and language-pair factor matrices, and d latent is the number of factors. Additionally, in order to incorporate the feature information about the language pairs as discussed in section 3.2, we incorporate Collective Matrix Factorization approach (Cortes, 2018). It incorporates the attribute information about items and/or users in the factorization algorithm by decomposing the language-pair feature matrix X  R |PT | n as LFT , such that L is shared across both decompositions. This helps to learn the latent representations for the pivot-language pairs from the task-wise performance as well as different linguistic and MMLM specific features 2 . In relation to Equation 1, we can think of task factors T to correspond to the task specific parameters  , language-pair factors L as the shared parameters  and the predictor model as f ( x p,t , t ;  , ) = ( TLT ) ( p,t ) , t . Both L and T are regularized seperately, but there is no group regularization term (  group = 0 ). Ye et al. (2021) also uses a Tensor Factorization approach for performance prediction which is similar to our CMF method. However, they train separate models for each task and factorize over metric specific attributes instead for a fine-grained prediction. Multi-Task Deep Gaussian Process Regression (MDGPR) : We use the multi-task variant of Gaussian Processes proposed in Bonilla et al. (2008) and utilize deep neural networks to define the kernel functions as in Deep GPs (Wilson et al., 2016). For comparison, we also report the scores of the single-task variant of this method which we denote as DGPR. See Appendix (section A.1) for details. Apart from these we also explore other multitask methods like Model Agnostic Meta Learning (MAML) (Finn et al., 2017), details of which we leave in the appendix (section A.1). In this section, we discuss our test conditions, datasets and training parameters for the different experiments. We consider two different test conditions: Leave One Language Out (LOLO) and Leave Low Resource Languages Out (LLRO). 2 Note that we can use a similar approach for providing side information for the tasks as well. Leave One Language Out : LOLO is a popular setup for multilingual performance prediction (Lauscher et al., 2020; Srinivasan et al., 2021), where for a given task, we choose a target language and move all of its instances from the prediction dataset to the test data. The models are then trained on the remaining languages and evaluated on the unseen test language. This is done for all the target languages available for a task, and the Mean Absolute Error (MAE) across languages is reported. In the multi-task setting we evaluate on one task at a time while considering the rest as helper tasks for which the entire data is used including the test language 3 . Leave Low Resource Languages Out : Through this evaluation strategy we try to emulate the real world use case where we only have test data available in high resource languages such as English, German and Chinese, and would like to estimate the performance on under-represented languages such as Swahili and Bengali. We use the language taxonomy provided by Joshi et al. (2020) to categorize the languages into six classes (0 = low to 5 = high) based on the number of resources available. We then move languages belonging to class 3 or below to our test set and train the models on class 4 and 5 languages only. Similar to LOLO, here too we allow the helper tasks to retain all the languages. We use the following 11 tasks provided in XTREME (Hu et al., 2020) and XTREME-R (Ruder et al., 2021) benchmarks: 1. Classification : XNLI (Conneau et al., 2018) , PAWS-X (Yang et al., 2019), and XCOPA (Ponti et al., 2020) 2. Structure Prediction : UDPOS (Nivre et al., 2018), and NER (Pan et al., 2017) 3. Question Answering : XQUAD (Artetxe et al., 2020), MLQA (Lewis et al., 2020), and TyDiQA-GoldP (Clark et al., 2020) 4. Retrieval : Tatoeba (Artetxe and Schwenk, 2019), Mewsli-X (Botha et al., 2020; Ruder et al., 2021), and LAReQA (Roy et al., 2020) All of these datasets have training data present 3 Note that this is a reasonable relaxation to make as it is closer to the real world use case where we would have the evaluation data for some languages in the standard tasks and would like to utilize that to make predictions on the same languages for the new ftask. For training XGBoost, we used 100 estimators with a maximum depth of 10. For Group Lasso, we used the implementation provided in the MuTaR software package 4 , and used a regularization strength of 0 . 01 . We optimized CMF's objective function using Alternating Least Squares (ALS), used 5 latent factors with a regularization parameter equal to 0 . 1 , and used the Collective Matrix Factorization python library 5 . In case of MDGPR, we used Radial Basis Function as the kernel and a two-layer MLP for learning latent features, with 50 and 10 units followed by ReLU activation. We set the learning rate and epochs as 0.01 and 200, and implemented it using GPyTorch 6 . Table 1 shows MAE (in %) for LOLO for different single-task and multi-task models on the tasks. For XLMR, we observe that multi-task models, primarily MDGPR, often outperform the best single-task models by significant margins, and for tasks like MewsliX we even see about 36% reduction in MAE. Overall, we see about 10% drop in LOLO errors on average for MDGPR compared to the best performing single-task model i.e. Lasso Regression. As expected, the benefit of multi-task 4 https://github.com/hichamjanati/mutar 5 https://github.com/david-cortes/ cmfrec 6 https://gpytorch.ai/ learning is even more prominent when we consider the tasks for which only a few (  10 ) data points are available. Here we see about 20% reduction in errors. For mBERT as well, we have similar observations, except that CMF performs slightly better than MDGPR. Note that the Average across task baseline is quite competitive and performs better than single-task XGBoost and MAML in average, and better than all models for LAReQA. Figure 2 plots the dependence of the number of helper tasks on the performance of the multi-task models. As expected, MAE decreases as helper tasks increase, especially for MDGPR and CMF. On a related note, the Pearson Correlation coefficient between MAE and number of tasks a target language is part of is found to be  0 . 39 , though the trend in this case is not as clear. Predicting the performance on low resource languages, for which often standard training and test datasets are not available, can be an important use case where multi-task performance prediction can be helpful. Figure 6 in appendix shows the classwise (Joshi et al., 2020) distribution of languages for the tasks that we consider in our experiments. As one would expect, for most tasks, test data is available for languages belonging to class-4 and class-5. Training performance prediction models without any task to transfer from can therefore, possibly lead to poor generalization on the low resource languages. On the other hand, for the same reason lack of test data, building accurate predictors for low-resource languages is necessary. MAE values for the LLRO evaluation setup are shown in figure 1 for XLMR. Results for mBERT follow similar trends and are reported in the Appendix (figure 7). For both XLMR and mBERT we observe that the three main multi-task models  Group Lasso, CMF and MDGPR  outperform the single-task models and baselines. Interestingly, for XLMR, the single task models XGBoost and Lasso perform even worse than the Average within Tasks baseline. Overall we see around 18% and 11% drop in MAE for Group Lasso over the best performing single-task model, for XLMR and mBERT respectively. An interesting consequence of zero-shot performance prediction is that the models can be directly used to infer the correlation (and possibly causation) between linguistic relatedness and pretraining conditions and zero-shot transferability. Multi-task learning, in this context, help us make more robust inferences, as the models are less prone to overfitting to a particular task or dataset. Figure 3 shows the SHAP values of the features for the Group Lasso model trained on XLMR's zero-shot performance data. As expected for Group Lasso, we see a block-sparsity behavior among the tasks. Features such as Rare Typological Traits (WMRR(t)), Tokenizer's Fertility (FERT ( t ) ) and Genetic Similarity ( s gen ( p, t ) ) are ignored in all the tasks. In contrast, for the single-task lasso regression (Figure 9 in Appendix), we see different sets of features selected for different tasks, which for the scale at which we operate, might not be indicative of the actual factors that affect the zero-shot performance in these tasks. Subword Overlap. Among the features that get selected for all tasks, we observe that Subword Overlap ( o sw ( p, t ) ) typically gets higher importance in XGB o o s t MAMLDGPRL a ss o AWTAATCMFG r o up L a ss o MDGPR 6 8 10 12 14 A v e r a g e LLROE rr o r 11.86 10.72 10.34 10.16 9.66 9.59 9.30 9.03 8.28 XLMR Figure 1: Leave Low Resource Out (LLRO) results for XLMR 1 2 3 4 5 6 7 8 9 10 Number of Helper Tasks 0.40.50.60.70.80.91.0 N o r m a li z e d LOLOE rr o r XLMR Model Group Lasso CMF MDGPR Figure 2: Number of helper tasks vs. LOLO MAE. retrieval (LAReQA and MewsliX) and sentence classification tasks (PAWS-X, XNLI). Since the retrieval tasks that we consider, as described in Ruder et al. (2021), measure the alignment between the cross lingual representations of semantically similar sentences, having a shared vocabulary between the languages can leak information from one to an-other (Wu and Dredze, 2019) which might improve the retrieval performance. Interestingly, if we compare this with the feature importance scores for the single task lasso model (Figure 9 in Appendix), we do see MewsliX task getting higher importance for the subword overlap, but LAReQA gets virtually zero SHAP value for this feature, showcasing how single-task models can misinterpret two similar tasks as requiring very different features. Our observation reinforce the generally held notion that vocabulary overlap between the pivot and target is beneficial for zero-shot transfer (Wu and Dredze, 2019), especially for retrieval tasks, though some studies have argued otherwise (Pires et al., 2019; K et al., 2020). (XQUAD and TyDiQA) tasks that require making predictions for each token in the input, we see that the tokenizer feature, PCW ( t ) , receive a higher SHAP value. In contrast, for single-task lasso, here too we do not observe high importance of this feature across these related tasks. Rust et al. (2021) note that languages such as Arabic where mBERT's multilingual tokenizer was found to be much worse than it's monolingual counterpart, there was a sharper drop in performance of mBERT compared to the monolingual model for QA, UDPOS and NER tasks than for sentiment classification. We believe that XLMR's surprisingly worse performance than mBERT for Chinese and Japanese on UDPOS might be correlated with it's significantly worse tokenizer for these languages based on the fertility (FERT) and Percentage Continued Words (PCW) feature values (see Appendix A.2 for exact values). The high SHAP values for PCW ( t ) further strengthen our belief 7 . Pre-training Size. Similar to the findings of Lauscher et al. (2020), we observe that pre-training corpus size has low SHAP value, and therefore, lower importance for lower level tasks such as UDPOS and NER, and higher SHAP values for higher level tasks like XNLI. Additionally, we extend their observations to tasks such as XCOPA, Tatoeba, MLQA and LAReQA where pre-training size seem to play a significant role in the performance prediction. Again, compared to single Lasso Regression model, we see a different selection pattern: Pre-training size receives a high SHAP value for UDPOS while for XNLI it is negligible. This neither fully conforms with our observations on the multi-task feature selections, nor with the previous work (Lauscher et al., 2020). Typological Relatedness Features. Out of all the typological relatedness features, we found Geographical Distance ( d geo ( p, t ) ) receiving highest SHAP values for all tasks, implying that geographical proximity between the pivot-target pair is an important factor in determining the zero-shot transferability between them. Lauscher et al. (2020) also observe positive correlations between geographical relatedness and zero-shot performance. The cross-task importance of geographic distance (unlike the other relatedness features) might be attributed to the 100% coverage across languages for the geo-7 Note that Rust et al. (2021) shows the importance of tokenizer metrics for the case where the multilingual models are fine-tuned on the target language, whereas we analyze their importance for zero-shot transfer. graphical vectors in the URIEL database. In contrast, Syntactic and Phonological vectors have missing values for a majority of the languages (Littell et al., 2017). Like Lauscher et al. (2020), we also see some dependence on syntactic ( s syn ( p, t ) ) and phonological ( s pho ( p, t ) ) similarities for XLMR's zero shot performance on XNLI and XQUAD tasks respectively. However, in both cases we found that the tokenizer feature PCW ( t ) receives a much higher SHAP value. Interestingly, genetic similarity ( s gen ( p, t ) ) is not selected for any task, arguably due to the block sparsity in feature selection of Group Lasso. We do see some tasks receiving high SHAP values for s gen ( p, t ) in single-task lasso (Figure 9 in Appendix). However, the number of such tasks as well as the SHAP values are on the lower side, implying that genetic similarity might not provide any additional information for zero-shot transfer over and above the geographical, syntactic and phonological similarities. Similar trends are observed in the case of mBERT as well (Figure 10 in appendix), with some minor differences. For instance, instead of PCW ( t ) , FERT ( t ) receives higher SHAP value; s syn ( p, t ) also receives higher importance, especially for tasks like UDPOS and XNLI, which is consistent with the findings of Lauscher et al. (2020). In this paper, we showed that the zero-shot performance prediction problem can be much more effectively and robustly solved by using multi-task learning approaches. We see significant reduction in errors compared to the baselines and single-task models, specifically for the tasks which have test sets available in a very few languages or when trying to predict the performance for low resource languages. Additionally, this approach allows us to robustly identify factors that influence zero-shot performance. Our findings in this context can be summarized as follows. 1. Subword overlap between the pivot and target has a strong positive influence on zero-shot transfer, especially for Retrieval tasks. 2. Quality of the target tokenizer , defined in terms of how often or how aggressively it splits the target tokens negatively influences zero-shot performance for word-level tasks such as POS tagging and Span extraction. 3. Pre-training size of the target positively 5461 influences zero-shot performance in many tasks, including XCOPA, Tatoeba, MLQA and LAReQA. 4. Geographical proximity between pivot and target is found to be uniformly important across all the tasks, unlike syntactic and phonological similarities, which are important for only some tasks. This last finding is especially interesting. As described earlier, geographical proximity is a more clear, noise-free and complete feature compared to the other relatedness metrics. However, one could also argue that since neighboring languages tend to have high vocabulary and typological feature overlap due to contact processes and shared areal features, geographical distance is an extremely informative feature for zero-shot transfer. Two direct implications of these findings are: (1) for effective use of MMLMs, one should develop resources in at least one pivot language per geographic regions, and (2) one should work towards multilingual tokenizers that are effective for most languages. There are a number of directions that can be explored in future related to our work. The prediction models can be extended to a multi-pivot and few-shot settings, as described in Srinivasan et al. (2021). Further probing experiments could be designed to understand the role of sub-word overlap on zero-shot transfer of Retrieval tasks. We would like to thank the LITMUS team at Microsoft for their valuable inputs and feedback over the course of this project.
abstain abstain method method method other abstain abstain abstain abstain abstain abstain abstain result abstain objective abstain result result method abstain objective abstain objective objective other other other other other other other other other other other other other other other other abstain other other other objective method abstain abstain abstain method method abstain method abstain other abstain abstain abstain method method method abstain method method abstain method abstain abstain abstain abstain abstain abstain abstain method method abstain method other method abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain method method method method abstain abstain method method method method method method method method method abstain method abstain method method abstain result result other result abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain result abstain result abstain abstain abstain abstain abstain abstain abstain abstain other abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain objective result method objective abstain abstain abstain abstain abstain abstain abstain abstain objective abstain abstain other