Unsupervised neural machine translation (UNMT) that relies solely on massive monolingual corpora has achieved remarkable results in several translation tasks. However, in real-world scenarios, massive monolingual corpora do not exist for some extremely low-resource languages such as Estonian, and UNMT systems usually perform poorly when there is not adequate training corpus for one language. In this paper, we first define and analyze the unbalanced training data scenario for UNMT. Based on this scenario, we propose UNMT self-training mechanisms to train a robust UNMT system and improve its performance in this case. Experimental results on several language pairs show that the proposed methods substantially outperform conventional UNMT systems. Recently, unsupervised neural machine translation (UNMT) that relies solely on massive monolingual corpora has attracted a high level of interest in the machine translation community (Artetxe et al., 2018; Lample et al., 2018a; Yang et al., 2018; Lample et al., 2018b; Wu et al., 2019; Sun et al., 2019, 2020b). With the help of cross-lingual language model pretraining (Lample and Conneau, 2019; Song et al., 2019; Sun et al., 2020a), the denoising auto-encoder (Vincent et al., 2010), and back-translation (Sennrich et al., 2016a), UNMT has achieved remarkable results in several translation tasks. However, in real-world scenarios, in contrast to the many large corpora available for high-resource languages such as English and French, massive monolingual corpora do not exist for some extremely low-resource languages such as Estonian. The UNMT system usually performs poorly in a low-resource scenario when there is not an adequate training corpus for one language. In this paper, we first define and analyze the unbalanced training data scenario for UNMT. Based on this scenario, we propose a self-training mechanism for UNMT. In detail, we propose self-training with unsupervised training (ST-UT) and self-training with pseudo-supervised training (ST-PT) strategies to train a robust UNMT system that performs better in this scenario. To the best of our knowledge, this paper is the first work to explore the unbalanced training data scenario problem in UNMT. Experimental results on several language pairs show that the proposed strategies substantially outperform conventional UNMT systems. In this section, we first define the unbalanced training data scenario according to training data size. Consider one monolingual corpus { X } in high-resource language L 1 and another monolingual corpus { Y } in low-resource language L 2 . The data size of { X } and { Y } are denoted by | X | and | Y | , respectively. In an unbalanced training data scenario, | X | is generally much larger than | Y | so that training data { X } is not fully utilized. To investigate UNMT performance in an unbalanced training data scenario, we empirically chose English (En)  French (Fr) as the language pair. The detailed experimental settings for UNMT are given in Section 5. We used a transformer based XLM toolkit and followed the settings of Lample and Conneau (2019). We randomly extracted 2 million sentences for each language from all 50 million sentences in the En and Fr training corpora to create small corpora and simulate unbalanced training data scenarios. Table 1 shows the UNMT performance for different training data sizes. The performance with 25M training sentences for both French and English configuration is similar to the baseline (50M training sentences for both French and English configuration). However, the UNMT performance decreased substantially (45 BLEU points) when the size of the training data decreased rapidly. In the unbalanced training data scenario, when training data for one language was added, they were not fully utilized and only slightly improved the UNMT's BLEU score. The performance (2M/50M) is similar with the UNMT system, configured 2M training sentences for both French and English. In short, Table 1 demonstrates that the UNMT performance is bounded by the smaller monolingual corpus. The UNMT model converges and even causes over-fitting in the low-resource language while the model in the high-resource language doesn't converge. This observation motivates us to better use the larger monolingual corpus in the unbalanced training data scenario. We first briefly describe the three components of the UNMT model (Lample and Conneau, 2019): cross-lingual language model pre-training, the denoising auto-encoder (Vincent et al., 2010), and back-translation (Sennrich et al., 2016a). Cross-lingual language model pre-training provides a naive bilingual signal that enables the back-translation to generate pseudo-parallel corpora at the beginning of the training. The denoising auto-encoder acts as a language model to improve translation quality by randomly performing local substitutions and word reorderings. Generally, back-translation plays an important role in achieving unsupervised translation across two languages. The pseudo-parallel sentence pairs produced by the model at the previous iteration are used to train the new translation model. The general back-translation probability is optimized by maximizing L bt = EX  P ( X ) EY  PMU  ( Y | X ) logP MU ( X | Y ) + EY  P ( Y ) EX  PMU  ( X | Y ) logP MU ( Y | X ) , (1) where P ( X ) and P ( Y ) are the empirical data distribution from monolingual corpora { X } , { Y } , and PMU ( Y | X ) and PMU ( X | Y ) are the conditional distributions generated by the UNMT model. In addition, MU  denotes the model at the previous iteration for generating new pseudo-parallel sentence pairs to update the UNMT model. Self-training proposed by Scudder (1965), is a semi-supervised approach that utilizes unannotated data to create better models. Self-training has been successfully applied to many natural language processing tasks (Yarowsky, 1995; McClosky et al., 2006; Zhang and Zong, 2016; He et al., 2020). Recently, He et al. (2020) empirically found that noisy self-training could improve the performance of supervised machine translation and synthetic data could play a positive role, even as a target. Based on these previous empirical findings and analyses, we propose a self-training mechanism to generate synthetic training data for UNMT to alleviate poor performance in the unbalanced training data scenario. The synthetic data increases the diversity of low-resource language data, further enhancing the performance of the translation, even though the synthetic data may be noisy. As the UNMT model is trained, the quality of synthetic data becomes better, causing less and less noise. Compared with the original UNMT model that the synthetic data is just used as the source part, we also use the synthetic data as the target part in our proposed methods. Newly generated synthetic data, together with original monolingual data, are fully utilized to train a robust UNMT system in this scenario. According to the usage of the generated synthetic training data, our approach can be divided into two strategies: ST-UT (Algorithm 1) and ST-PT (Algorithm 2). ST-UT: In this strategy, we first train a UNMT model on the existing monolingual training data. The final UNMT system is trained using the STUT strategy for k 1 epochs. For one epoch l in the ST-UT strategy, a subset { X sub } is selected randomly from monolingual training data { X } . The quantity of { X sub } is (cid:15) of | X | , (cid:15) is a quantity Algorithm 1 ST-UT strategy Input: Monolingual training data { X } , { Y } 1: Train a UNMT model MU 0 on monolingual training data { X } , { Y } 2: while epoch l  max epoch k 1 do 3: Select a subset { X sub } randomly on monolingual training data { X } 4: Apply the last trained UNMT model M Ul  1 to this subset { X sub } to generate synthetic data { Y subM } = { M Ul  1 ( X sub ) } 5: Train a new UNMT model M Ul on monolingual data { X } , { Y } and synthetic data { Y subM } 6: end while Output: The final translation model M Uk 1 ratio hyper-parameter. The last trained UNMT model M Ul  1 is used to generate synthetic data { Y subM } = { M Ul  1 ( X sub ) } . The synthetic data are used 1 , together with the monolingual data to train a new UNMT model M Ul . Therefore, the translation probability for the ST-UT strategy is optimized by maximizing L bt = EX  P ( X ) EY  PMU  l ( Y | X ) logP M Ul ( X | Y ) + EY  P ( Y ) EX  PMU  l ( X | Y ) logP M Ul ( Y | X ) + EY  PMU  l  1 ( Y | X ) EX  PMU  l ( X | Y ) logP M Ul ( Y | X ) , (2) where PM Ul ( Y | X ) and PM Ul ( X | Y ) are the conditional distribution generated by the UNMT model on epoch l for the ST-UT strategy and PMU  l  1 ( Y | X ) is the conditional distribution generated by the UNMT model on epoch l  1 for the ST-UT strategy. ST-PT: In this strategy, we first train a UNMT system on the existing monolingual training data and switch to a standard neural machine translation system from UNMT system with synthetic parallel data for both translation directions. The final translation system is trained using the ST-PT strategy for k 2 epochs. For one epoch q in the ST-PT strategy, a subset { X sub } is selected randomly from monolingual training data { X } , and all monolingual data { Y } is selected. The quantity of { X sub } is (cid:15) of | X | , (cid:15) is a quantity ratio hyper-parameter. The last trained pseudo-supervised neu-1 In contrast to using all synthetic data, we tried to train a language model and select more fluent synthetic data according to a language model perplexity score. This did not improve translation performance. Algorithm 2 ST-PT strategy Input: Monolingual training data { X } , { Y } 1: Train a UNMT model MU 0 on monolingual training data { X } , { Y } 2: while epoch q  max epoch k 2 do 3: Select a subset { X sub } randomly on monolingual training data { X } and all monolingual training data { Y all } 4: Apply the last trained PNMT model M Pq  1 ( MP 0 = MU 0 ) to generate { Y subM } = { M Pq  1 ( X sub ) } and { X allM } = { M Pq  1 ( Y all ) } 5: Train a new PNMT model M Pq on synthetic parallel corpora { X sub , Y subM } and { Y all , X allM } 6: end while Output: The final translation model M Pk 2 ral machine translation (PNMT) model 2 M Pq  1 is used to generate { Y subM } = { M Pq  1 ( X sub ) } and { X allM } = { M Pq  1 ( Y all ) } to create synthetic parallel data { X sub , Y subM } and { Y all , X allM } . Note that we use the UNMT model to generate synthetic parallel data during the first epoch of the ST-PT strategy. Synthetic parallel data { X sub , Y subM } and { Y sub , X subM } are selected to train a new PNMT model M Pq that can generate translation in both directions. Therefore, the translation probability for ST-PT strategy is optimized by maximizing L bt = EX  P ( X ) EY  PMP  q  1 ( Y | X ) logP M Pq ( X | Y ) + EX  P ( X ) EY  PMP  q  1 ( Y | X ) logP M Pq ( Y | X ) + EY  P ( Y ) EX  PMP  q  1 ( X | Y ) logP M Pq ( Y | X ) + EY  P ( Y ) EX  PMP  q  1 ( X | Y ) logP M Pq ( X | Y ) , (3) where PM Pq ( Y | X ) and PM Pq ( X | Y ) are the conditional distributions generated by the PNMT model on epoch q for the ST-PT strategy; PMP  q  1 ( Y | X ) and PMP  q  1 ( X | Y ) are the conditional distributions generated by the PNMT model on epoch q  1 for the ST-PT strategy. We considered three language pairs in our simulation experiments: FrEn, Romanian (Ro)En and
abstain abstain objective objective abstain abstain abstain abstain abstain objective objective objective objective abstain abstain abstain abstain abstain abstain abstain method method abstain abstain abstain abstain abstain abstain abstain abstain abstain other other other other other other other other other abstain abstain abstain abstain abstain method abstain abstain abstain other abstain abstain other abstain abstain abstain abstain abstain abstain other abstain abstain other abstain