Unsupervised machine translation, which utilizes unpaired monolingual corpora as training data, has achieved comparable performance against supervised machine translation. However, it still suffers from data-scarce domains. To address this issue, this paper presents a novel meta-learning algorithm for unsupervised neural machine translation (UNMT) that trains the model to adapt to another domain by utilizing only a small amount of training data. We assume that domain-general knowledge is a significant factor in handling data-scarce domains. Hence, we extend the meta-learning algorithm, which utilizes knowledge learned from high-resource domains, to boost the performance of low-resource UNMT. Our model surpasses a transfer learning-based approach by up to 2-3 BLEU scores. Extensive experimental results show that our proposed algorithm is pertinent for fast adaptation and consistently outperforms other baselines. Unsupervised neural machine translation (UNMT) leverages unpaired monolingual corpora for its training, without requiring an already labeled, parallel corpus. Recently, the state of the art in UNMT (Conneau and Lample, 2019; Song et al., 2019; Ren et al., 2019) has achieved comparable performances against supervised neural machine translation (NMT) approaches. In contrast to supervised NMT, which uses a parallel corpus, training the UNMT model requires a significant number of monolingual sentences (e.g., 1M-3M sentences). However, the prerequisite limits UNMT's applicability to low-resource domains, especially for  equal contributions  This work done in NAVER Corp. domain-specific document translation tasks. Since gathering or creating those documents requires domain specific knowledge, the monolingual data themselves are scarce and expensive. In addition, the minority languages (e.g., Uzbek and Nepali) make the problem of data scarcity even worse. Yet, UNMT for low-resource domains is not an actively explored field. One naive approach is to train a model on high-resource domains (e.g., econ-omy and sports) while hoping the model will generalize on an unseen low-resource domain (e.g., medicine). However, recent studies have shown that non-trivial domain mismatch can significantly cause low translation accuracy on supervised NMT tasks (Koehn and Knowles, 2017). Another reasonable approach is transfer learningparticularly, domain adaptationwhich has shown performance improvements in the supervised NMT literature (Freitag and Al-Onaizan, 2016; Zeng et al., 2019). In this approach, the model is first pretrained using data from existing domains and then finetuned on a new domain. However, this approach can suffer from overfitting and catastrophic forgetting due to a small amount of training data and a large domain gap. As an effective method for handling a small amount of training data, meta-learning has shown its superiority in various NLP studies such as dialog generation, machine translation, and natural language understanding (Qian and Yu, 2019; Gu et al., 2018; Dou et al., 2019). In general, the meta-learning approach is strongly affected by the number of different tasks where tasks are defined as languages or domains from the aforementioned studies. However, in practice, the previous studies may struggle to gather data to define tasks because they rely on a supervised model that requires labeled corpora. In this respect, we argue that applying a meta-learning approach to the unsupervised model is more feasible and achievable than the supervised model because it can define multiple different tasks with unlabeled corpora. Therefore, we introduce a new meta-learning approach for UNMT, called MetaUMT, for low-resource domains by defining each task as a domain. The objective of MetaUMT is to find the optimal initialization for the model parameters that can quickly adapt to a new domain even with only a small amount of monolingual data. As shown in Fig. 1 (a), we define two different training phases, a meta-train and a meta-test phase, and simulate the domain adaption process to obtain optimally initialized parameters. Specifically, the meta-train phase adapts model parameters to a domain while the meta-test phase optimizes the parameters obtained from the meta-train phase. After obtaining optimally initialized parameters through these two phases, we fine-tune the model using a target domain (i.e., a low-resource domain). Although the initial parameters optimized through MetaUMT are suitable for adapting to a low-resource domain, these parameters may not fully maintain the knowledge of high-resource domains. Concretely, in the meta-test phase, MetaUMT optimizes initial parameters using the adapted parameters; however, it discards meta-train knowledge used to update adapted parameters in the meta-train phase. Therefore, instead of validating the same domain used in the meta-train phase, we intend to inject generalizable knowledge into the initial parameters by utilizing another domain in the meta-test phase. This prevents overfitting from the data scarcity issue. As shown in Fig. 1 (b), we propose an improved meta-learning approach called MetaGUMT for low-resource UNMT by explicitly infusing common knowledge across multiple source domains as well as generalizable knowledge from one particular domain to another. In other words, we do not only encourage the model to find the optimally initialized parameters that can quickly adapt to a target domain with low-resource data, but also encourage the model to maintain common knowledge (e.g., general words such as determiners, conjunctions, and pronouns) which is obtainable from multiple source domains. Furthermore, due to a small amount of training data in a low-resource domain, the model can suffer from overfitting; however, we attempt to handle overfitting by leveraging generalizable knowledge that is available from one domain to another. Our proposed meta-learning approach demonstrates consistent improvements over other baseline models. Overall, our contributions can be summarized as follows:  We apply a meta-learning approach for UNMT. To the best of our knowledge, this is the first study to use a meta-learning approach for UNMT, where this approach is more suitable to a UNMT task than a supervised one. We empirically demonstrate that our enhanced method, MetaGUMT, shows fast convergence on both pre-training (i.e., meta-learning with source domains) and finetuning (i.e., adapting to a target domain). The model trained with MetaGUMT consistently outperforms all baseline models including MetaUMT. This demonstrates that finding optimally initialized parameters that incorporate high-resource domain knowledge and generalizable knowledge is significant in handling a low-resource domain. Our study leverages two components from the natural language processing (NLP) domain: low-resource NMT and meta-learning. In this section, we discuss previous studies by concentrating on these two main components. Based on the success of attention-based models (Luong et al., 2015; Vaswani et al., 2017), NMT obtains significant improvement in numerous language datasets, even showing promising results (Wu et al.) in different datasets. However, the performance of NMT models depends on the size of the parallel dataset (Koehn and Knowles, 2017). To address this problem, one conventional approach is utilizing monolingual datasets. Recent studies point out the difficulty of gathering parallel data, whereas the monolingual datasets are relatively easy to collect. To facilitate monolingual corpora, several studies apply dual learning (He et al., 2016), back-translation (Sennrich et al., 2016b), and pretraining the model with bilingual corpora (Hu et al., 2019; Wei et al., 2020). Furthermore, as a challenging scenario, recent studies propose the UNMT methods without using any Meta-train parallel corpora (Lample et al., 2018a; Artetxe et al., 2018; Yang et al., 2018). The UNMT models show comparable performances by extending the back-translation method (Conneau et al., 2018) and incorporating methods such as shared Byte Pair Encoding (BPE) (Lample et al., 2018b) and cross-lingual representations (Conneau and Lample, 2019), following those of the supervised NMT. However, since these approaches require plenty of monolingual datasets, they suffer in a low-resource domain. Transferring the knowledge from high-resource domains to a low-resource domain is one alternative way to address this challenge. A few studies concentrate on transferring the knowledge from the rich-resource corpora into the low-resource one. Several models (Chu and Wang, 2018; Hu et al., 2019) show better performances than when trained with the low-resource corpora only. However, these approaches are applicable in specific scenarios where one or both of the source and target domains consist of a parallel corpus. To address the issues, we define a new task as the unsupervised domain adaptation on the low-resource dataset. Our work is more challenging than any other previous studies, since we assume that both the low-resource target domain and the source domain corpora are monolingual. Given a small amount of training data, most of the machine learning models are prone to overfitting, thus failing to find a generalizable solution. To handle this issue, meta-learning approaches seek for how to adapt quickly and accurately to a low-resource task, and show impressive results in various domains (Finn et al., 2017; Javed and White, 2019). The meta-learning approaches aim to find the optimal initialization of the model parameters that adapts the model to a low-resource dataset in a few iterations of training (Finn et al., 2017; Ravi and Larochelle, 2016). Owing to the success of the meta learning, recent studies apply the meta learning to low-resource NMT tasks, including multilingual NMT (Gu et al., 2018) and the domain adaptation (Li et al., 2020). These studies assume that all the training corpora consist of the parallel sentences. However, a recent work (Li et al., 2018) utilizes the meta learning approach to find a generalized model for multiple target tasks. However, it is not focused on adapting a specific target task since its main goal is to handle the target task without using any low-resource data. Our study attempts to address the low-resource UNMT by exploiting meta-learning approaches. Moreover, we present two novel losses that encourage incorporating high-resource knowledge and generalizable knowledge into the model parameters. Our proposed approaches show significant performance improvements in adapting to a low-resource target domain. In this section, we first introduce the notation of the general UNMT models. We then describe the three steps for the UNMT task: initialization, language modeling, and back-translation. On these three steps, we illustrate how each step contributes to improving the performance of UNMT. Notations . We denote S and T as a source and a target monolingual language dataset. x and y represent the source and the target sentences from S and T . We assume the NMT model is parame-Choosing a domain Sampling another domain English Sentences English Sentences English Sentences English Sentences Meta-test phase Metatrain phase (E) Update : (C) Obtain by using one-step gradient : (D) Compute each loss : (B) (A) English Sentences English Sentences English Sentences English Sentences English Sentences English Sentences English Sentences English Sentences English Sentences English Sentences Figure 2: Overall training process of our proposed MetaGUMT. terized by  . We also denote M s  s and M t  t as language models in a source and a target language, respectively, while denoting M s  t and M t  s as the machine translation models from the source to the target language and vice versa. Initialization . A recent UNMT model (Lample et al., 2018b) is based on a shared encoder and decoder architecture for the source and the target language. Due to the shared encoder and decoder for each language, initializing the model parameters of the shared encoder and decoder is an important step for competitive performances (Conneau et al., 2018; Lample et al., 2018a; Artetxe et al., 2018; Yang et al., 2018). Conneau and Lample (2019) propose the XLM (cross-lingual language model) to initialize parameters, showing significantly improved performances for UNMT. Among various initialization methods, we leverage the XLM as our initialization method. Language modeling . We use a denoising auto-encoder (Vincent et al., 2008) to train the UNMT model, reconstructing an original sentence from a noisy one in a given language. The objective function is defined as follows: L lm = E x  S [  log M s  s ( x | C ( x ))]+ E y  T [  log M t  t ( y | C ( y ))] , (1) where C is a noise function described in (Lam-ple et al., 2018b), which randomly drops or swaps words in a given sentence. By reconstructing the sentence from the noisy sentence, the model learns the language modeling in each language. Back-translation . Back-translation helps the model learn the mapping functions between the source and the target language by using only the monolingual sentences. For example, we sample a sentence x and y from source language S and target language T . To make pseudo-pair sentences from the sampled source sentence, we deduce the target sentence from the source sentence, such that y (cid:48) = M s  t ( x ) , resulting in the pseudo parallel sentence, i.e., ( x, y (cid:48) ) . Similarly, we obtain ( x (cid:48) , y ) , where x (cid:48) is the translation of a target sentence, i.e., M t  s ( y ) . We do not back-propagate when we generate the pseudo-parallel sentence pairs. In short, the back-translation objective function is L bt = E y  T [  logM s  t (cid:0) y | x (cid:48) (cid:1) ]+ E x  S [  logM t  s (cid:0) x | y (cid:48) (cid:1) ] . This section first explains our formulation of a low-resource unsupervised machine translation task where we can apply a meta-learning approach. Afterwards, we elaborate our proposed methods, MetaUMT and MetaGUMT. We utilize the meta-learning approach to address a low-resource challenge for unsupervised machine translation. Moreover, we extend MetaUMT into MetaGUMT to explicitly incorporate learned knowledge from multiple domains. Finn et al. (2017) assume multiple different tasks to find the proper initial parameters that can quickly adapt to a new task using only a few training examples. In this paper, we consider tasks in the meta-learning as domains, where D out = {D 0 out , ..., D nout } represents n out-domain datasets (i.e., source domain datasets), and D in indicates an in-domain dataset (i.e., a target domain dataset), which can be the dataset in an arbitrary domain not included in D out . Each domain in both D out and D in is assumed to be composed of unpaired language corpora, and we create D in as a low-resource monolingual dataset 1 . To adapt our model to the low-resource in-domain data, we finetune the UNMT model by minimizing both the losses described in Eqs. (1) and (2) with D in . In order to obtain an optimal initialization of the model parameters, allowing the model to quickly adapt to a new domain with only a small number of monolingual training data, MetaUMT uses two training phases, the meta-train phase and the meta-test phase. During the meta-train phase, the model first learns domain-specific knowledge by updating initial model parameters  to temporary model parameters  i , i.e., adapted parameters. Then, in the meta-test phase, the model learns the adaptation by optimizing  with respect to  i . From the domain adaption perspective, two phases simulate the domain adaption process. The model first adapts to a specific domain through the meta-train phase, and this adaption is evaluated in the meta-test phase. Meta-train phase . We obtain  i for each ith out-domain dataset by using one-step gradient descent, i.e.,  i =      L s D i out (  ) , (3) where L s D i out is represented as L s D i out = L lm D i out (  ) + L bt D i out (  ) . (4) D iout is the i -th out-domain dataset, and  is the learning rate for the meta-train phase. As previously discussed in Section 3, the language modeling and back-translation losses are essential in facilitating the unsupervised machine translation. Hence, L s consists of L lm and L bt , where each loss function is computed with D iout . Meta-test phase . The objective of the meta-test phase is to update  using each  i learned from the 1 We randomly sample the 5,000 tokens (  300 sentences) from the in-domain training dataset. meta-train phase by using each L s D iout (cid:48) 2 . We call this update as a meta-update, defined as        n (cid:88) i =0 L s D iout (cid:48) (  i ) , (5) where  is another learning rate in the meta-test phase. Since Eq. (5) requires the second-order gradient, the equation is simplified with the first-order gradient by replacing the second-order term. Finn et al. (2017) showed that the first-order approximation of the meta-learning maintains the performance while minimizing the computational cost. To handle a data scarcity issue from a meta-learning perspective, it is critical to be able to make the initialized model to adapt to a data-scarce domain. However, since a small amount of training data in the new domain may cause the model to overfit and prevent utilizing high-resource domain knowledge, it is important to incorporate high-resource domain knowledge and generalizable knowledge into the model parameters. To address this issue, we extend the existing meta-learning approach via two novel losses, which we call an aggregated meta-train loss and a cross-domain loss. The former contributes to incorporating high-resource domain knowledge into the model parameters, while the latter encourages our model, after trained using a particular domain, to still generalize well to another domain, i.e., cross-domain generalization. Meta-train phase . As shown in Fig. 2 (C), via Eqs. (3) and (4), we obtain  i from each ith out-domain datasets. Since this phase is exactly same with the meta-train phase of MetaUMT, we leave out the details. Meta-test phase . The aggregated meta-train loss, which refers to Fig. 2 (D), is computed using all out-domain datasets, i.e., L ag = n (cid:88) i =0 L s D iout (  ) . This loss term allows the model to learn the source domain knowledge that is potentially applicable to a target domain. Moreover, to alleviate the overfitting after adapting to the low-resource domain, we 2 L s D iout and L s D iout (cid:48) indicate different batch sampled data from same D i . introduce a cross-domain loss, which is in Fig. 2 (D), as L cd = n (cid:88) i =0 L s D icd (  i ) , (7) where L s D icd = L s D iout (cid:48) (  i )+ L s D iother (  i ) , i.e., computing the cross-domain loss with the data from D iout (cid:48) as well as those from other domains D iother . To obtain the optimal initialization  for model parameters, we define our total loss function, which is Fig. 2 (E), as the sum of the two of our losses, i.e.,        ( L cd + L ag ) . In summary, our aggregated meta-train and cross-domain losses encourage our model to accurately and quickly adapt to an unseen target domain. The overall procedure is described in Algorithm A.1. This section first introduces experiment settings and training details. Afterwards, we show empirical results in various scenarios. We conduct our experiments on eight different domains 3 (Appendix T.2). Each domain dataset is publicly available on OPUS 4 (Tiedemann, 2012). We utilize the eight domains for out-domain ( D out ) and in-domain datasets ( D in ). To build the monolingual corpora of in-domain and out-domain datasets, we sample data from the parallel corpus. We made sure to include at most one sentence from each pair of parallel sentences. For instance, we sample the first half of the sentences as unpaired 3 Acquis (Law), EMEA (Medical), IT, Tanzil (Koran), Subtitles, EUbookshop (EUB), Europarl, and GlobalVoices (GV) 4 http://opus.nlpl.eu/ source data and the other half as truly unpaired target data. Consequently, the sampled monolingual corpora contain no translated sentence in each language. Each of the two monolingual corpora contains the equal number of sentences for each language (e.g., English and German). For our low-resource scenarios, we sample 5,000 tokens from a selected in-domain corpus for each language. Note that the out-domain dataset represents the full monolingual corpora. As our base model, we use a Transformer (Vaswani et al., 2017), which is initialized by a masked language model from XLM (Conneau and Lample, 2019) using our out-domain datasets. All the models consist of 6 layers, 1,024 units, and 8 heads. UNMT model is trained with only the in-domain monolingual data, composed of 5,000 words for each language. Supervised neural machine translation model (NMT) is trained with in-domain parallel datasets, which we arrange in parallel with the two in-domain monolingual corpora. Unadapted model is pretrained with only the out-domain datasets and evaluated on the in-domain datasets. Transfer learning model is a finetuned model, which is pretrained with the out-domain datasets and then finetuned with a low-resource in-domain dataset. it utilizes both in-domain and out-domain datasets for finetuning. That is, the training batch is sampled evenly from in-domain and out-of-domain datasets. In order to verify that leveraging the high-resource domains (i.e., the source domains) effects to handle the low-resource domains (i.e., the target domain), we compare the unsupervised and supervised models with ours and other baseline models. As shown in Table 1, the unsupervised model trained on in-domain data suffers from data scarcity because it only uses low-resource in-domain data. Although the unsupervised and supervised models are initialized by XLM, those models show the worst performance in all the cases. This result indicates that when the small size of an in-domain corpus is given, it is appropriate to utilize the out-domain datasets rather than to train only with low-resource data. In addition, the performance of the unadapted model is far behind compared to other models, such as the mixed finetuned model, transfer learning model, MetaUMT, and MetaGUMT. This implies that we need an adequate strategy of leveraging the high-resource domains to improve the performance. We further compare the performance between our proposed approaches (i.e., MetaUMT and MetaGUMT) and the other two finetuning models (i.e., the transfer learning and the mixed finetuned ones). Our methods exhibit the leading performances in both directions of translation ( en  de ), and consistently achieve improvements of 23 BLEU score in most of settings. Furthermore, MetaGUMT consistently obtains better BLEU scores and converges faster than MetaUMT. We assert that our proposed losses (i.e., the aggregated meta-train and the cross-domain losses) help the model not only to perform well even on the unseen in-domain dataset but also to accelerate the convergence speed. As shown in Fig. 3 (A), we compare our proposed methods with the transfer learning approach by varying the sizes of an in-domain monolingual corpus. The smaller the size of training data is, the wider the performance gap between the two approaches and the transfer learning model becomes. It means that meta-learning is an effective approach to alleviate the performance degradation, preventing the model from overfitting to the low-resource data. Compared to the transfer learning model, MetaUMT demonstrates a better performance than other methods in various settings. However, MetaGUMT exhibits even better performances consistently in all settings owing to our proposed losses (Eq. (8)). The transfer learning approach shows the worst performance except for the unadapted model, even though it exploits the in-domain corpus after being pretrained with the out-domain datasets. Additionally, we analyze the number of iterations required for a model to converge given an in-domain dataset. As shown in Fig. 3 (B), the meta-learning approaches rapidly converge after only a few iterations, even faster than the transfer learning one does. As the number of in-domain training words increases, the transfer learning approach requires a much larger number of iterations until convergence than our meta-learning approaches. It can be seen that MetaUMT and MetaGUMT rapidly adapt to an unseen domain. Moreover, owing to the encapsulated knowledge from the high-resource do-Parameter Initial  Finetuned  DD out D in Unseen Meidcal Law Koran EUB IT GV Subtitles Europarl De-En En-De De-En En-De De-En En-De De-En En-De De-En En-De De-En En-De De-En En-De De-En En-De Transfer 30.98 26.96 34.8 30.28 13.72 11.59 12.32 10.01 20.98 17.74 17.4 14.25 10.92 9.18 22.31 16.58 Mixed finetuned ------11.77 9.96 22.84 16.92 MetaUMT 33.0 23.39 27..74 15.4 4.89 0.79 6.78 2.59 9.45 4.68 2.77 1.06 12.95 10.58 23.91 18.7 MetaGUMT 37.37 31.63 42.73 37.3 18.2 13.84 13.72 11.8 24.0 19.24 21.24 17.38 13.45 10.89 24.44 19.31 Table 2: BLEU scores evaluated on out-domain and in-domain data with initial  and finetuned  , respectively. mains, MetaGUMT converges within a relatively earlier iteration than MetaUMT does. In summary, the meta-learning-based methods quickly converge in the low-resource domain, improving the performances over the transfer learning method in various low-resource settings. This indicates that the meta-learning-based approaches are suitable to alleviate the data deficiency issue in scarce domains. Furthermore, our losses in Eq. (8) enhance the capabilities of aggregating domain general knowledge and finding adequate initialization. An advantage of our meta-learning approaches is that they can find an optimal initialization point from which the model can quickly adapt to a low-resource in-domain dataset. The transfer learning model requires twice more iterations until convergence than ours does. As shown in Fig. 3 (C), MetaUMT and MetaGUMT not only converge quickly but also outperform the other baseline methods. Specifically, compared to MetaUMT, MetaGUMT is effective in achieving an optimized initialization at an earlier iteration. These results indicate that our additional losses (i.e., the cross-domain and aggregated meta-train losses) are ben-eficial in boosting up the ability for finding an optimal initialization point when training the model with the out-domain datasets. We assume that the domain generalization ability and high-resource domain knowledge are helpful for the UNMT model to translate the low-resource domain sentences. First, to identify whether the model encapsulates the high-resource knowledge from multiple sources, we evaluate our model on out-domain datasets (i.e., D out ) with initial  . As shown in Table. 2, MetaGUMT shows remarkable performances over MetaUMT in all domains, even better than the transfer learning models. In other words, MetaUMT demonstrates poor performances Cross-domain Aggregated meta-train De-En En-De Average  (cid:55) (cid:55) 27.09 24.6 25.85 (cid:88) (cid:55) 27.37 24.76 26.06 +0.21 (cid:55) (cid:88) 27.54 24.90 26.22 +0.37 (cid:88) (cid:88) 27.85 25.06 26.46 +0.61 Table 3: Effectiveness of each cross-domain and aggregated meta-train loss. in D out , compared to MetaGUMT. This can be explained as MetaGUMT uses an aggregated meta-train loss such that MetaGUMT is able to encapsulate the high-resource domain knowledge. As shown in Table. 1, MetaGUMT achieves superior performances, showing that MetaGUMT is capable of leveraging the encapsulated knowledge when finetuning the low-resource target domain. Secondly, our cross-domain loss encourages the model to have a generalization capability after adapting to the low-resource target domain. As shown in Unseen column in Table. 2, MetaGUMT outperforms the other models. It can be seen that our model has the domain generalization ability after the finetuning stage due to the cross-domain loss in the meta-test phase. In UNMT, data unbalancing is often the case in that source language (e.g., English) data are abundant and the target language (e.g., Nepali) data are scarce (Kim et al., 2020). We extend our experiment to the unbalanced scenarios to examine whether our proposed model shows the same tendency. In this scenario, the low-resource target domain dataset consists of monolingual sentences from one side with two times more tokens than the monolingual sentences from the other. As shown in Table. 4, MetaGUMT outperforms in all unbalanced data cases. It shows that MetaGUMT is feasible to a practical UNMT scenario where the number of sentences is different in the source and target languages. The only difference against the main experiment setting 5.1 is the condition that the in-domain corpus is unbalanced. We also include the # tokens Mixed MetaUMT MetaGUMT En De En-De De-En En-De De-En En-De De-En 5k 10k 26.04 31.90 28.80 32.65 29.43 34.28 8k 16k 26.09 32.01 27.84 32.93 29.62 34.39 16k 32k 26.44 32.37 27.92 32.96 30.10 34.44 32k 64k 27.39 32.84 28.67 33.52 29.83 34.77 Table 4: Results on the unbalanced monolingual Law domain data during the finetuning stage, where D out is GV, Euorparl, EUB, Subtitles, Medical and Koran. result of the transfer learning model in Table. T.4. We empirically show the effectiveness of the cross-domain and aggregated meta-train losses, as shown in Table 3 5 . First, compared to MetaUMT which does not use any of the two losses, incorporating the cross-domain loss improves the average BLEU score by 0.21. The cross-domain loss acts as a regularization function that prevents the model from overfitting during the finetuning stage. Second, the aggregated meta-train loss, another critical component of our model, allows the model to utilize the high-resource domain knowledge in the finetuning stage. This also improves the average BLEU score by 0.37 from MetaUMT. Lastly, combining both cross-domain and aggregated meta-train losses significantly enhances the result in both directions of translation ( En  De ), indicating that they are complementary to each other. We examine how the performances change against the different number of source domains for each approach. As shown in Table. 5 6 , MetaGUMT consistently outperforms the transfer, the mixed-finetune, and MetaUMT approaches. As the size of the source domains increases, so does the performance gap between ours and the transferring based models, i.e., transferring and mixed-finetune models. This indicates that the meta-learning based approaches are highly effected by the size of the domains in the meta-train phase, and also, if the number of source domains is large enough to capture the general knowledge, the meta-learning based approaches are suitable to handle the low-resource target task (i.e., machine translation in a low-resource domain). 5 The models are pretrained on Subtitles, Law, EUB, Europarl, IT, and GV and then finetuned on the Medical data. 6 The 4 case contains the Medical, Law, Koran and EUB domains. 5 and 6 additionally utilize one more domain(i.e., IT) and two more domains(i.e.,IT and GV), respectively. # D out MetaGUMT MetaUMT Transfer Mixed En-De De-En En-De De-En En-De De-En En-De De-En 4 5.97 7.47 5.87 7.24 5.75 7.17 5.87 7.22 5 7.58 9.49 7.33 9.01 7.17 8.08 7.20 8.68 6 10.89 13.45 10.58 12.95 9.18 10.92 9.96 11.77 Table 5: Effectiveness of the different number of source domains between meta-learning based approaches and the transfer learning approach, where # D out represents the number of out-domain datasets in the pretraining stage. This paper proposes a novel meta-learning approach for low-resource UNMT, called MetaUMT, which leverages multiple source domains to quickly and effectively adapt the model to the target domain even with a small amount of training data. Moreover, we introduce an improved method called MetaGUMT, which enhances cross-domain generalization and maintains high-resource domain knowledge. We empirically show that our proposed approach consistently outperforms the baseline methods with a nontrivial margin. We believe that our proposed methods can be extended to semi-supervised machine translation as well. In the future, we will further analyze other languages, such as Uzbek and Nepali, instead of languages like English and German. This work was partially supported by Institute of Information & communications Technology Planning & Evaluation (IITP) grant funded by the Korea government (MSIT) (No. 2019-0-00075, Artificial Intelligence Graduate School Program (KAIST) and No. 2020-0-00368, A Neural-Symbolic Model for Knowledge Acquisition and Inference Techniques) and by the National Research Foundation of Korea (NRF) grant funded by the Korean government (MSIT) (No. NRF-2019R1A2C4070420) References Mikel Artetxe, Gorka Labaka, Eneko Agirre, and Kyunghyun Cho.
abstain abstain objective result objective result objective abstain abstain abstain objective abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method objective abstain abstain result abstain result abstain abstain method abstain abstain objective result abstain objective objective objective objective objective objective abstain abstain other other other other other other other other other other other other other objective method other other other other other other other abstain objective objective objective abstain objective other abstain other other other abstain other other other other abstain other method other other other other method abstain abstain method other method method method method abstain abstain method method method abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain method abstain method abstain abstain abstain abstain abstain method method abstain abstain method method abstain abstain abstain method other method method abstain abstain result abstain abstain abstain abstain abstain other abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain result abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain other abstain abstain abstain abstain abstain result abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain result abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain other objective result objective objective method other other