The commonly used framework for unsupervised machine translation builds initial translation models of both translation directions, and then performs iterative back-translation to jointly boost their translation performance. The initialization stage is very important since bad initialization may wrongly squeeze the search space, and too much noise introduced in this stage may hurt the final performance. In this paper, we propose a novel retrieval and rewriting based method to better initialize unsupervised translation models. We first retrieve semantically comparable sentences from monolingual corpora of two languages and then rewrite the target side to minimize the semantic gap between the source and retrieved targets with a designed rewriting model. The rewritten sentence pairs are used to initialize SMT models which are used to generate pseudo data for two NMT models, followed by the iterative back-translation. Experiments show that our method can build better initial unsupervised translation models and improve the final translation performance by over 4 BLEU scores. Recent work has shown successful practices of unsupervised machine translation (UMT) (Artetxe et al., 2017; Lample et al., 2017, 2018; Artetxe et al., 2018b; Marie and Fujita, 2018; Ren et al., 2019; Lample and Conneau, 2019). The common framework is to build two initial translation models (i.e., source to target and target to source) and then do iterative back-translation (Sennrich et al., 2016a; Zhang et al., 2018) with pseudo data generated by each other. The initialization stage is important because bad initialization may wrongly squeeze the search space, and too much noise introduced in this stage may hurt the final performance. Previous methods for UMT (Lample et al., 2018; Artetxe et al., 2018b; Marie and Fujita, 2018; Ren et al., 2019) usually use the following n-gram embeddings based initialization. They first build phrase translation tables with the help of unsupervised cross-lingual n-gram embeddings (Conneau et al., 2017; Artetxe et al., 2018a), and then use them to build two initial Phrase-based Statistical Machine Translation (PBSMT) (Koehn et al., 2003) models with two language models. However, there are two problems with their initialization methods. (1) Some complex sentence structures of original training sentences are hard to be recovered with the n-gram translation tables. (2) The initial translation tables inevitably contain much noise, which will be amplified in the subsequent process. In this paper, we propose a novel retrieve-and-rewrite initialization method for UMT. Specifically, we first retrieve semantically similar sentence pairs from monolingual corpora of two languages with the help of unsupervised cross-lingual sentence embeddings. Next, with those retrieved similar sentence pairs, we run GIZA++ (Och and Ney, 2003) to get word alignments which are used to delete unaligned words in the target side of the retrieved sentences. The modified target sentences are then rewritten with a designed sequence-to-sequence rewriting model to minimize the semantic gap between the source and target sides. Taking the pairs of the source sentences and corresponding rewritten targets as pseudo parallel data, we then build two initial PBSMT models (source-to-target and target-to-source), which are used to generate pseudo parallel data to warm up NMT models, followed by an iterative back-translation training process. Our code is released at https://github.com/Imagist-Shuo/RRforUNMT.git. Our contributions are threefold. (1) We propose a novel method to initialize unsupervised MT models with a retrieve-and-rewrite schema, which can Figure 1: Method overview. (2) We design an effective seq-to-seq architecture based on the Transformer to rewrite sentences with semantic constraints. (3) Our method significantly outperforms the previous non-pre-training based UMT results on en fr and en de translation tasks, and give the first unsupervised en zh translation results on WMT17. Our method can be divided into three steps as shown in Figure 1. First, we do similar sentences retrieval (  2.1) from two monolingual corpora with the help of unsupervised cross-lingual sentence embeddings. Next, to minimize the semantic gap between the source and retrieved targets, we do target sentences rewriting (  2.2) by deleting unaligned words in the target side, and generate complete and better-aligned targets via our rewriting model with the help of missing information provided by the source. After that, we treat the rewritten pairs as the pseudo parallel data for translation models initialization and training (  2.3). Given two monolingual corpora D x and D y of two languages X and Y respectively, we first build unsupervised cross-lingual word embeddings of X and Y using fastText (Bojanowski et al., 2017) and vecmap (Artetxe et al., 2018a), and then we obtain cross-lingual sentence embeddings based on the cross-lingual word embeddings via SIF (Arora et al., 2017). After that, we use the marginal-based scoring (Artetxe and Schwenk, 2018) to retrieve Figure 2: Example of rewriting. Examples retrieved from monolingual English and Chinese corpora are shown in Figure 1 in the Appendix A. 2.2 Target Sentences Rewriting As shown in Figure 2, having retrieved similar sentence pairs { x, y } , we first run GIZA++ (Och and Ney, 2003) on these pairs and obtain the word alignment information. Then, for each target sentence y , we remove the unaligned words from it according to lexical translation probabilities of GIZA++ output. We replace each deleted word with (cid:104) DEL (cid:105) in y to get the incomplete target sentence y (cid:48) . Meanwhile, we record the unaligned words in the source as x m 1 where m is the number of the unaligned source words. Next, we feed y (cid:48) and x m 1 into a sequence-to-sequence model to generate the refined target sentence  y . The rewritten pairs { x,  y } are 1 For each source sentence, we choose 30 nearest neighbors in the target language, which have approximately similar lengths to the source (within the difference of  5 words), and keep the neighbors with the scores more than 0.6. Our rewriting model is a modification of Transformer (Vaswani et al., 2017) shown as Figure 3. We initialize the embedding layer of the second input part with pre-trained cross-lingual word embeddings because its content should be independent of languages. We keep it fixed during training. Thus the second part is like a memory recording semantic information of words. We concatenate the readout embeddings of both parts with a separator, and feed them to the Transformer encoder, so that the attention mechanism will take effect on both parts together. For model training , due to the lack of references, we need to build training data for the rewriting model from monolingual corpus D y . Firstly, we remove 20 to 30 percent of words from a given sentence y  D y , and replace them with (cid:104) DEL (cid:105) to get y (cid:48) . Next, we randomly swap contiguous words in y (cid:48) with the probability of 0.2 to introduce some noises. Then we record the removed words as set s m 1 and randomly drop/add some words from/to this set. We then treat y (cid:48) and s m 1 as the inputs, and y as the output to train the model. For model inference , we feed the incomplete sentence y (cid:48) and unaligned source words x m 1 into the trained model and generate the refined sentence  y . Note there seems to be a bias between the training and inference that s m 1 during training are in the same language as y , while during inference, they are from the source language X . But the bias has been eliminated since the second input part of the encoder is the readout cross-lingual embeddings, which is independent of languages. Once we get { x,  y } generated above, we use them to train initial PBSMT models, and use the SMT models to produce pseudo data to setup two NMT models, followed by the iterative back-translation. In our experiments, we consider three language pairs, English-French ( en fr ), English-German ( en de ) and English-Chinese ( en zh ). For en , fr and de , we use 50 million monolingual sentences in NewsCrawl from 2007 to 2017. As for zh , we use the Chinese side from WMT17 en zh parallel data. 2 For the convenience of comparison, we use newstest 2014 as the test set for en fr , newstest 2016 for en de , and newstest 2017 for en zh . The data preprocessing is described in Appendix D. Baselines Our method is compared with eight baselines of unsupervised MT systems listed in the upper area of Table 1. The first three baselines are unsupervised NMT models, and the fourth baseline is an unsupervised PBSMT model. The fifth baseline is an extract-and-edit schema for unsupervised neural machine translation. The sixth and seventh baselines are hybrid models of NMT and PBSMT. And the last baseline is a pre-training based method. The comparison results are reported in Table 1. From the table, we find that our method significantly outperforms the best non-pre-training based baseline with an average of 4.63 BLEU scores on all pairs. Note that Lample and Conneau (2019) is based on pre-training, which uses much more monolingual data than our method. Even so, we reach comparable results on the en fr pair. 2 Note that we only retrieve similar sentences from sampled 20 million sentences in each monolingual corpus and use Hierarchical Navigable Small World (HNSW) (Malkov and Yashunin, 2018) to build embedding index for space and time efficiency. During the iterative back-translation process in  2.3, we use the whole monolingual corpora. three baselines initialize their SMT models with phrase tables inferred from n-gram embeddings and language models. From the table, we find that our proposed method gives better initialization to SMT models. Even the SMT models trained with only the retrieved sentences reach higher performance than previous methods, which verifies that the noise within the retrieved sentences is random to a greater extent and can be easily eliminated by SMT models, which is consistent with Khayrallah and Koehn (2018). With the target sentences rewritten by our rewriting model, the quality of extracted phrases can be further improved. We also try to directly train NMT models with the rewritten pseudo data, but only get the BLEU scores under 10, which means there is still much noise for SMT to eliminate in the pseudo pairs. We build two test sets to quantify the performance of our rewriting models. The first test set denoted as in-domain , is from our synthetic training data. As described before, we build training samples using monolingual data according to the rules in  2.2. We select 8M sentences from the monolingual corpus of a certain language for model training and randomly sample 8k sentences as development and test sets respectively. In addition, we also test our rewriting model on newstest2014 ( en fr ), which is denoted as out-domain . We first run GIZA++ on the parallel sentences in the original test set to find the golden alignments between source and target words. Next, we randomly delete up to 30% words in the target side and record their aligned source words. Then we feed the incomplete target sentence and the recorded source words into our model to recover the original target. The BLEU scores on both test sets are listed in Table 3, which shows our rewriting model has good performance. Unsupervised machine translation becomes a hot research topic in recent years. The pioneering methods are based on NMT models (Transformer) (Artetxe et al., 2017; Lample et al., 2017; Yang et al., 2018) trained with denoising auto-encoder (Vincent et al., 2010) and iterative back-translation. The following work shows that SMT methods and the hybrid of NMT and SMT can be more effective (Artetxe et al., 2018b; Lample et al., 2018; Marie and Fujita, 2018; Ren et al., 2019; Artetxe et al., 2019). They build the initial PBSMT models with language models and phrase tables inferred from unsupervised cross-lingual n-gram embeddings. Recently, Lample and Conneau (2019) propose a pre-training method and achieve state-of-the-art performance on unsupervised en fr and en de translation tasks. But they use much more monolingual data from Wikipedia than previous work and this paper. We must also mention the work of Wu et al. (2019). They similarly use retrieval and rewriting framework for unsupervised MT. However, ours is different from theirs in two aspects. First, we efficiently calculate the cross-lingual sentence embeddings via a training-free method SIF rather than a pre-trained language model. Second, our rewriting method is based on the word alignment information which is more explicit than their max pooling, and our rewriting model is more simple but effective so that the rewriting results can be directly used without extra training techniques. In this paper, we propose a novel method for unsupervised machine translation with a retrieve-and-rewrite schema. We first retrieve similar sentences from monolingual corpora and then rewrite the targets with a rewriting model. With the pseudo parallel data, we better initialize PBSMT models and significantly improve the final iteration performance as the experiments show. This work is supported in part by National Key R&D Program of China AAA0102301 , and NSFC 61925203 & U1636210 & 61421003 .
abstain abstain objective objective abstain result abstain abstain abstain abstain abstain abstain abstain abstain objective objective method abstain method other objective objective method objective method method method method abstain method abstain method method abstain method method abstain method abstain abstain method method method abstain method method method abstain abstain method method method method method method abstain abstain abstain abstain abstain result abstain result method method abstain abstain abstain abstain method method abstain method abstain abstain abstain abstain abstain result other other other other other method abstain other objective abstain objective objective result other