The overreliance on large parallel corpora sig-nificantly limits the applicability of machine translation systems to the majority of language pairs. Back-translation has been dominantly used in previous approaches for unsupervised neural machine translation, where pseudo sentence pairs are generated to train the models with a reconstruction loss. However, the pseudo sentences are usually of low quality as translation errors accumulate during training. To avoid this fundamental issue, we propose an alternative but more effective approach, extract-edit , to extract and then edit real sentences from the target monolingual corpora. Furthermore, we introduce a comparative translation loss to evaluate the translated target sentences and thus train the unsupervised translation systems. Experiments show that the proposed approach consistently outperforms the previous state-of-the-art unsupervised machine translation systems across two benchmarks (English-French and English-German) and two low-resource language pairs (English-Romanian and English-Russian) by more than 2 (up to 3 . 63 ) BLEU points. Promising results have been achieved in Neural Machine Translation (NMT) by representation learning (Cho et al., 2014b; Sutskever et al., 2014). But recent studies (Koehn and Knowles, 2017; Isabelle et al., 2017; Sennrich, 2017) highlight the overreliance of current NMT systems on large parallel corpora. In real-world cases, the majority of language pairs have very little parallel data, so the models need to leverage monolingual data to address this challenge (Gulcehre et al., 2015; Zhang and Zong, 2016; He et al., 2016; Yang et al., 2018). est approaches (Lample et al., 2018a; Artetxe et al., 2018; Lample et al., 2018b) focus on the fully unsupervised scenario. Back-translation has been dominantly used in these approaches, where pseudo sentence pairs are generated to train the translation systems with a reconstruction loss. However, it is inefficient because the generated pseudo sentence pairs are usually of low quality. During the dual learning of back-translation, the errors could easily accumulate and thus the learned target language distribution would gradually deviate from the real target distribution. This critical drawback hinders the further development of the unsupervised NMT systems. An alternative solution is to extract real parallel sentences from comparable monolingual corpora, and then use them to train the NMT systems. Recently, neural-based methods (Chu et al., 2016; Grover and Mitra, 2017; Gregoire and Langlais, 2018) aim to select potential parallel sentences from monolingual corpora in the same domain. However, these neural models need to be trained on a large parallel dataset first, which is not applicable to language pairs with limited supervision. In this paper, we propose a radically different approach for unsupervised NMT extract-edit , a powerful alternative to back-translation (see Figure 1). Specifically, to train the source-to-target translation model, we first extract potential parallel sentence candidates in the target language space given a source language sentence. Since it cannot be guaranteed that there always exist potential parallel sentence pairs in monolingual corpora, we further propose a simple but effective editing mechanism to revise the extracted sentences, making them aligned with the source language sentence. Then a comparative translation loss is introduced to evaluate the translated sentence based on the extracted-and-edited ones and train the translation model. Compared to back-translation, extract-edit avoids the distribution deviation issue by extracting and editing real sentences from the target language space. Those extracted-and-edited sentences serve as pivotal points in the target language space to guide the unsupervised learning. Thus, the learned target language distribution could be closer to the real one. The extract-edit model and the translation model, the two major parts of our method, can be jointly trained in a fully unsupervised way. Empirical results on popular benchmarks show that exact-edit consistently outperforms the state-of-the-art unsupervised NMT system (Lample et al., 2018b) with back-translation across four different languages pairs. In summary, our main contributions are three-fold 1 :  We propose a more effective alternative paradigm to back-translation, extract-edit, to train the unsupervised NMT systems with potentially real sentence pairs;  We introduce a comparative translation loss for unsupervised learning, which optimizes the translated sentence by maximizing its relative similarity with the source sentence among the extracted-and-edited pairs;  Our method advances the previous state-of-the-art NMT systems across four different 1 The source code can be found in this repository: https://github.com/jiaweiw/ Extract-Edit-Unsupervised-NMT language pairs under monolingual corpora only scenario. Without parallel sentence pairs as constraints on mapping language spaces, training NMT systems is an ill-posed problem because there are many potential mapping solutions. Nevertheless, some promising methods have been proposed in this field (Lample et al., 2018a; Artetxe et al., 2018; Lample et al., 2018b). The main technical protocol of these approaches can be summarized as three steps: Initialization , Language Modeling , and Back-Translation . In this section, we mainly introduce the three steps and the crucial settings that we have followed in our work. In the remainder of the paper, we denote the space of source and target languages by S and T , respectively. enc and dec refer to the encoder and decoder models in the sequence-to-sequence systems. V s  t stands for the composition of enc in the source language and dec in the target language, which can be viewed as the source-to-target translation system. Initialization Given the ill-posed nature of the unsupervised NMT task, a suitable initialization method can help model the natural priors over the mapping of two language spaces we expect to reach. There are mainly two initialization methods: (1) bilingual dictionary inference (Conneau et al., 2018; Artetxe et al., 2018; Lample et al., 2018a) and (2) byte-pair encoding (BPE) (Sen-nrich et al., 2016b; Lample et al., 2018b). As shown in Lample et al. (2018b), the inferred bilingual dictionary can provide a rough word-by-word alignment of semantics, and the BPE can reduce the vocabulary size and eliminate the presence of unknown words in the output results. In our extract-edit approach, to extract potential parallel sentence pairs, we need to compare the semantic similarity of sentences between two languages first. A proper initialization can also help align the semantic spaces and extract potential parallel pairs within them. Thus, following the previous methods, we use the inferred bilingual dictionary as described in Conneau et al. (2018) for unrelated language pairs and the shared BPE in Lample et al. (2018b) as initialization for related ones. can train language models on both source and target languages. These models express a data-driven prior about the composition of sentences in each language. In NMT, language modeling is accomplished via denosing autoencoding, by minimizing: L lm (  enc ,  dec ) = E x S [  log V s  s ( x | C ( x ))]+ E y T [  log V t  t ( y | C ( y ))] (1) where C is a noise model with some words dropped and swapped,  enc and  dec are the learnable parameters of enc and dec . V s  s and V t  t are the encoder-decoder language models on the source and target sides, respectively. In our extract-edit approach, we follow similar settings and adopt the noise model proposed by Lample et al. (2018a). Note that the parameters of all enc are shared (same for dec ) in our framework to ensure a strong alignment and mapping between two languages. This sharing operation is essential for both the translation model and the extract-edit model. Thus, we use enc to represent encoders in source language modeling enc S and in target language modeling enc T (same for dec ). work to train the unsupervised NMT system. It couples the source-to-target translation model with a backward target-to-source model and trains the whole system with a reconstruction loss. This can be viewed as converting the unsupervised problem into a supervised scenario by generating pseudo language pairs (He et al., 2016). Despite the popularity of back-translation in the previous methods (Lample et al., 2018a; Artetxe et al., 2018; Lample et al., 2018b), we argue that it suffers from the low-quality pseudo language pairs. Thus, in this work, we propose a new paradigm, extract-edit, to address this issue by extracting and editing potential real parallel sentences. Below we describe our approach in details. The overview of our extract-edit approach is shown in Figure 2. We first extract and edit real sentences from the target language space according to their similarities with the source sentence. These extracted-and-edited sentences serve as pivotal points in the target language space, which locate a probable region where the real target sentence could be. Then we introduce a comparative translation loss to evaluate the translated sentence and train the system. Basically, the comparative translation loss encourages the translated sentence to approximate the real sentence by maximizing its relative similarity with the source sentence compared to the extracted-and-edited sentences. As a result, we manage to minimize the deviation of the learned target language distribution and the mapping noises between two language spaces. Most existing methods in comparable corpora mining introduce two encoders to represent sentences of two languages separately, and then use another network to measure the similarity (Chu et al., 2016; Grover and Mitra, 2017; Gregoire and Langlais, 2018). However, owing to the shared encoders and decoders in language modeling, the semantic spaces of two languages are already strongly connected in our scenario. Therefore, to avoid extra computation resources, we directly use the enc in language modeling to obtain sentence embeddings for two languages. As shown in Figure 2 (a), for a given source sentence s , we use the nearest neighbor search based on L 2 distance to find topk real sentences from the target language space ( k is a hyper-parameter decided empirically). The sentence embeddings used for searching are computed based on the shared encoder enc . The reason to choose topk sentences rather than top1 is to keep a high recall rate and obtain more related samples from the target language space. Finally, given the source sentence s , we denote M as a set of the k potential parallel target sentences: M = { t | min 1 ,  ,k ( || e s  e t || ) , t  T } , (2) where e s and e t are sentence embeddings encoded by the shared encoder enc . Even though the extracted sentences could serve as pivotal points to guide NMT, there is no guarantee that there always exists a parallel sentence in the target corpus. Thus, in order to make it closer to the real paired sentence in the target language space, we propose an editing mechanism to revise the extracted target sentence t  M based on the semantics of the source sentence s . As described in Figure 2 (b), we employ a maxpooling layer to reserve the more significant features between the source sentence embedding e s and the extracted sentence embedding e t ( t  M ), and then decode it into a new sentence t (cid:48) : M (cid:48) = { t (cid:48) | t (cid:48) = dec ( maxpooling ( e s , e t )) , t  M } , (3) where M (cid:48) is the set of the extracted-and-edited sentences. Based on the semantic information of the source sentence s , we can further improve the extracted results with this editing mechanism. Unlike other studies using the editing to generate more structural sentences (Guu et al., 2018; Hashimoto et al., 2018), here the revised sentences are designed to serve as better pivotal points in the target language space to guide the translation procedure. This can also be viewed as adding constraints when aligning the two language spaces. Given a source sentence s , we can translate it as t  using the source-to-target translation model P s  t . Meanwhile, a set M (cid:48) of k sentences can also be generated by the extract-edit approach described above. Although the M (cid:48) may contain potential parallel sentences t (cid:48) for s , we cannot directly use ( s, t (cid:48) ) as ground-truth sentence pairs to train the translation model V s  t because the NMT system is sensitive to noises (Cho et al., 2014a; Cheng et al., 2018). The rough operation like this will result in sub-optimal translation performance. Therefore, in order to assess the quality of the translated sentence t  and train the translation model V s  t , we introduce an evaluation network R for evaluating the relative similarities between the source and target sentences among all sentence pairs. The evaluation network R is a multilayer perceptron; it takes the target sentence embedding e t and source sentence embedding e s as inputs, and converts them into the joint embedding space as r t and r s . So the similarity  ( t | s ) = cosine ( r t , r s ) = r t  r s || r t |||| r s || . Then, a softmax-like formulation is used to compute the ranking score for the translated sentence t  given the extracted-and-edited sentence set M (cid:48) where the hyper-parameter  is similar to the inverse temperature of the softmax function. Lower  encourages the model to treat all extracted-edited sentences equally, while higher  highlights the importance of sentences with higher-score. Comparative Translation As introduced above, the ranking score calculates the relative similarity between the < s, t  > pair and all the extracted-and-edited pairs < s, t (cid:48) > . Assuming we have a good evaluation network R with  R denoting its parameters, we further introduce the comparative translation loss L com for unsupervised machine translation: L com (  enc |  R ) =  E (log P ( t  = V s  t ( s ) | s, M (cid:48) )) , (6) where  enc is the parameters of the shared encoder enc . Basically, the translation model is trying to minimize the relative distance of the translated sentence t  to the source sentence s compared to the topk extracted-and-edited sentences in the target language space. Intuitively, we view the topk extracted-and-edited sentences as the anchor points to locate a probable region in the target language space, and iteratively improve the source-to-target mapping via the comparative learning scheme. Combined with the language modeling constraints as described in Equation 1, the final loss function for training the the translation model V s  t is defined as: L s  t (  enc ,  dec |  R ) =  lm L lm (  enc ,  dec )+  com L com (  enc |  R ) , (7) where  lm and  com are hyper-parameters weighing the importance of the language modeling and the comparative learning. Adversarial Objective Meanwhile, we need to learn a good evaluation network R to transform sentence embedding of the shared encoder into the comparable space. The evaluation network R is also shared by two languages to ensure a strong connection between two language spaces. Inspired by adversarial learning (Goodfellow et al., 2014), we can view our translation system as a genera-tor that learns to generate a good translation with a higher similarity score than the extracted-and-edited sentences, and the evaluation network R as a discriminator that learns to rank the extracted-and-edited sentences (real sentences in the target language space) higher than the translated sentences. Thus, we have the following objective function for the evaluation network R : LR (  R ) =  E t (cid:48)  M (cid:48) (log P ( t (cid:48) | s, M (cid:48) )) . Based on Equation 7 and 8, the final adversarial objective is defined as min  enc , dec max  RL (  enc ,  dec ,  R ) = L R (  R ) + L s  t (  enc ,  dec |  R ) , (9) where the translation model V s  t and the evaluation network R play the two-player mini-max game. We evenly alternately update between the encoder-decoder translation model and the evaluation network. The detailed training procedure is described in Algorithm 1. In the fully unsupervised setting, we do not have access to parallel sentence pairs. Thus, we need to find a criterion correlated with the translation quality to select hyper-parameters. For a neural translation model V s  t , we propose the following criterion D s  t to tune the hyper-parameters: D s  t = E s S [ E (log P ( t  | s, M (cid:48) ))] , (10) where t  = V s  t ( s ) . Basically, we choose the hyper-parameters with the maximum expectation of the ranking scores of all translated sentences. We consider four language pairs: English-French ( en-fr ), English-German ( en-de ), English-Russian ( en-ru ) and English-Romanian ( en-ro ) for evaluation. For a fair comparison, we use the same corpora as in Lample et al. (2018b) for these languages for fair comparison. For English, French, Model en  fr fr  en en  de de  en LSTM Cell Lample et al. (2018b) 24.28 (+0 . 00) 23.74 (+0 . 00) 14.71 (+0 . 00) 19.60 (+0 . 00) Ours (Top1 Extract) 24.43 (+0 . 15) 23.90 (+0 . 16) 14.54 (  0 . 17) 19.49 (  0 . 11) Ours (Top1 Extract + Edit) 24.54 (+0 . 26) 24.08 (+0 . 34) 14.63 (  0 . 08) 19.57 (  0 . 03) Ours (Top10 Extract) 26.12 (+1 . 84) 25.83 (+2 . 09) 17.01 (+2 . 30) 21.40 (+1 . 80) Ours (Top10 Extract + Edit) 26.97 (+2 . 69) 26.66 (+2 . 92) 17.48 (+2 . 77) 21.93 (+2 . 33) Transformer Cell Lample et al. (2018b) 25.14 (+0 . 00) 24.18 (+0 . 00) 17.16 (+0 . 00) 21.00 (+0 . 00) Ours (Top1 Extract) 25.30 (+0 . 16) 24.23 (+0 . 05) 17.12 (  0 . 04) 21.06 (+0 . 06) Ours (Top1 Extract + Edit) 25.44 (+0 . 30) 24.36 (+0 . 18) 17.14 (  0 . 02) 21.10 (+0 . 10) Ours (Top10 Extract) 26.91 (+1 . 77) 25.64 (+1 . 46) 19.11 (+1 . 95) 22.84 (+1 . 84) Ours (Top10 Extract + Edit) 27.56 (+2 . 42) 26.90 (+2 . 72) 19.55 (+2 . 39) 23.29 (+2 . 29) Model en  ro ro  en en  ru ru  en LSTM Cell Lample et al. (2018b) 19.65 (+0 . 00) 18.52 (+0 . 00) 6.24 (+0 . 00) 7.83 (+0 . 00) Ours (Top1 Extract) 19.73 (+0 . 08) 18.56 (+0 . 04) 6.32 (+0 . 08) 7.99 (+0 . 16) Ours (Top1 Extract + Edit) 19.81 (+0 . 16) 18.69 (+0 . 17) 6.44 (+0 . 20) 8.12 (+0 . 29) Ours (Top10 Extract) 21.57 (+1 . 92) 20.32 (+1 . 80) 8.87 (+2 . 63) 9.76 (+1 . 93) Ours (Top10 Extract + Edit) 22.08 (+2 . 43) 20.83 (+2 . 31) 9.35 (+3 . 11) 10.21 (+2 . 38) Transformer Cell Lample et al. (2018b) 21.18 (+0 . 00) 19.44 (+0 . 00) 7.98 (+0 . 00) 9.09 (+0 . 00) Ours (Top1 Extract) 21.15 (  0 . 03) 19.52 (+0 . 08) 8.03 (+0 . 05) 9.20 (+0 . 11) Ours (Top1 Extract + Edit) 21.23 (+0 . 05) 19.59 (+0 . 15) 8.16 (+0 . 18) 9.28 (+0 . 19) Ours (Top10 Extract) 23.04 (+1 . 86) 21.43 (+1 . 99) 10.24 (+2 . 26) 12.29 (+3 . 20) Ours (Top10 Extract + Edit) 23.31 (+2 . 13) 21.60 (+2 . 16) 11.07 (+3 . 09) 12.72 (+3 . 63) Table 1: The experimental results on all four language pairs and directions. German and Russian, all the available sentences are used from the WMT monolingual News Crawl datasets from years 2007 through 2017. As for Romanian, we combine the News Crawl dataset and WMT'16 monolingual dataset. The translation results are evaluated on newstest 2014 for en-fr , and newstest 2016 for en-de , en-ro and en-ru . We follow previous methods (Koehn et al., 2007; Lample et al., 2018b) to initialize our models. Initialization We use Moses scripts (Koehn et al., 2007) for tokenization. While the system requires cross-lingual BPE embeddings to initialize the shared lookup table for related languages, we set the number of BPE codes as 60 , 000 . Following the previous preprocessing protocol (Lample et al., 2018b), the embeddings are then generated using fastText (Bojanowski et al., 2017) with an embedding dimension of 512, a context window of size 5 and 10 negative samples. Model Structure In this work, the NMT models can be built upon long short-term memory (LSTM) (Hochreiter and Schmidhuber, 1997) and Transformer (Vaswani et al., 2017) cells. For LSTM cells, both the encoder and decoder have 3 layers. As for Transformer, we use 4 layers both in the encoder and the decoder. As for both LSTM and Transformer, all encoder parameters are shared across two languages. Similarly, we share all decoder parameters across two languages. Both two model structure are optimized using Adam (Kingma and Ba, 2014) with a batch size of 32 . The rate for LSTM cell is 0 . 0003 while Transformer's is set as 0 . 0001 . The weights in Equation 7 are  lm =  ext = 1 . The  for calculating ranking scores is 0 . 5 . As for the evaluation network R , we use a multilayer perceptron with two hidden layers of size 512 . For efficient nearest neighbor search in the extracting step, we use the open-source Faiss library (Johnson et al., 2017) 2 . We calculate the similarity of sentences in 2 https://github.com/facebookresearch/ faiss each episode instead of each batch for computational efficiency. In this study, we aim to validate the effectiveness of extract-edit versus back-translation for unsupervised neural machine translation (NMT), so we set the unsupervised NMT method in Lample et al. (2018b) as the baseline because it currently achieves the state-of-the-art performance on all language pairs. 3 The overall translation results across four language pairs are shown in Table 1. In most of the cases, our proposed extract-edit approach can outperform the baseline models trained with back-translation. Our full models (LSTM/Transformer + Top10 Extract + Edit) achieve more than 2 BLEU points improvement consistently across all the language pairs. Especially, on the ru  en translation with the Transformer cell, our full model surpasses the baseline score by 3 . 63 BLEU points. These results validate the effectiveness of our approach and indicate that the proposed extract-edit learning framework can learn a better mapping and alignment between language spaces than back-translation. However, if extracting only top1 target sentence in our approach, the performances are not always improved (e.g., en  de , de  en , and en  ro ). Besides, Top-10 Extract + Edit models consistently outperforms Top-1 Extract + Edit . This is because more extracted-and-edited sentences lead to a higher recall, so more useful information will be used to guarantee the translation quality. The comparative translation loss can avoid the model suffering from the noise while taking advantage of more information. In other words, it is more likely to project the source sentence into the probable region in the target language space with more sentences serving as the anchor points, and the comparative learning scheme iteratively approximates towards more accurate target points. This highlights the importance of the extraction number k , which we further discuss next. 3 Note that for a fair comparison, we are not including the results of the unsupervised phrase-based statistical machine translation system (PBSMT). But theoretically, our extract-edit learning framework can be generalized to other types of machine translation systems such as PBSMT. The Effect of Extraction Number k As shown in Table 1, the number k of the extracted-and-edited sentences plays a vital role in our approach. Thus for a more intuitive overview of its impact, we further train and evaluate multiple models with k = 1 , 3 , 5 , 8 , 10 on en  fr translation task. The detailed results are shown in Figure 3. This study shows that the translation performance of our approach is indeed improving as k increases. Because as we analyze above, larger k ensure a higher recall and thus more critical semantic information can be utilized to assist the translation. Besides, the diversity of extracted-and-edited sentences can potentially provide a more accurate localization of the probable region where the target sentence should be. Although we can infer that the models would perform even better with k > 10 from Figure 3, more computational resources will be required for that and we already observe a decelerated growth of BLEU scores from k = 8 to k = 10 . Therefore, in this paper, we set k = 10 for the full models. The Quality of Extraction Model In this section, we quantitatively evaluate the unsupervised extraction part of our model and compare it with the state-of-the-art supervised extraction model. Following Gregoire and Langlais (2018), we train a fully supervised parallel pair extraction model, where two Bi-LSTMs are implemented to encode sentences of two languages, and a feed-forward network is followed to culminate in a sigmoid output layer. The model is trained with around 500 , 000 English-French parallel sentence pairs sampled from Europarl corpus (Koehn, 2005). As for our unsupervised extraction model, we directly use the jointly trained extraction part in our frame-Noise Model Hits@1 Hits@3 Hits@5 Hits@8 Hits@10 Hits@15 Hits@20 0% Supervised (Upperbound) 67.3 80.7 89.9 94.5 97.1 98.7 99.3 Unsupervised (Ours) 52.2 54.6 68.8 80.2 89.1 91.8 93.3 50% Supervised (Upperbound) 64.8 78.0 86.8 91.3 95.6 97.4 99.0 Unsupervised (Ours) 46.9 49.7 62.1 73.4 83.2 87.6 89.2 90% Supervised (Upperbound) 63.7 76.4 84.2 89.1 93.8 96.5 98.1 Unsupervised (Ours) 41.5 46.8 58.0 69.3 77.2 83.9 87.8 Table 2: The experimental results of parallel sentence mining on the newstest 2012 en  fr translation dataset with different levels of added sentence noises. work to extract the potential parallel sentences based on the scores computed by Equation 4. For evaluation, we sample 1 , 000 parallel sentences from the newstest 2014 corpus and create three test sets with a noise ratio 0%, 50%, and 90% to simulate noisy real-world data. We report Hits@ k results, which shows the percentage of the golden parallel sentences appear within the topk place. The detailed results are shown in Table 2. Although our extraction model structure is different from the supervised extraction model, it can still give us a good insight into the upperbound and gap of performance. We can observe a noticeable gap between unsupervised and supervised methods, but the gap is narrowing as the rank increases. Meanwhile, in our unsupervised method, the performance grows quickly when k  10 . From Table 2 we also notice that k = 10 is a sweet point, where the accuracy is high and the computational cost is relatively acceptable. The Effect of Comparative Translation Finally, we aim to roughly evaluate the effect of the proposed comparable translation loss in our model. Thus, we compare our model with a two-staged NMT system, where we extract and edit the parallel pairs and retrain the NMT system with the standard maximum likelihood estimation (MLE) loss in a supervised way (by taking the extracted-and-edited sentences as the ground-truth targets). We compare the performance on the en  fr newstest 2014 dataset, and the results are shown in Table 3. We can observe that with the MLE loss, the translation performance will drop nearly 50% . The results indirectly reflect that the NMT systems are sensitive to noises in the training datasets. Meanwhile, it demonstrates by treating extracted-edit sentences as pivotal points instead of ground truth, our proposed comparative translation loss can avoid the NMT model suffers from the noise. Although our extract-edit approach can achieve better performance than the back-translation mechanism, it is still worth mentioning that our approach has more strict constraints on the domains of the source and target corpus. The extract-edit approach will work well when there is information overlap in the two language spaces. When there is little overlap in terms of domains, it will be much harder to find a good cluster of initial candidates, which may also complicate the editing process. As for the back-translation mechanism, it requires less overlap in terms of the language spaces because the language priors can be learnt in any domains. However, the corpus with matching domains can be easily obtained nowadays (e.g., Wikipedia and the news articles), which makes our extract-edit approach still widely applicable. Unsupervised NMT The current NMT systems (Sutskever et al., 2014; Cho et al., 2014a; Bahdanau et al., 2015; Gehring et al., 2017; Vaswani et al., 2017) are known to easily over-fit and result in an inferior performance when the training data is limited (Koehn and Knowles, 2017; Isabelle et al., 2017; Sennrich, 2017). Many research efforts have been spent on how to utilize the monolingual data to improve the NMT system when only limited supervision is available (Gulcehre et al., 2015; Sennrich et al., 2016a; He et al., 2016; Zhang and Zong, 2016; Yang et al., 2018). Recently, Lample et al. (2018a); Artetxe et al. (2018); Lample et al. (2018b) make encouraging progress on unsupervised NMT structure mainly based on initialization, denoising language modeling, and back-translation. However, all these unsupervised models are based on the back-translation learning framework to generate pseudo language pairs for training. Our work leverages the information from real target language sentences. Comparable Corpora Mining Comparable corpora mining aims at extracting parallel sentences from comparable monolingual corpora such as news stories written on the same topic in different languages. Most of the previous methods align the documents based on metadata and then extract parallel sentences using human-defined features (Munteanu and Marcu, 2002, 2006; Hewavitharana and Vogel, 2011). Recent neural-based methods (Chu et al., 2016; Grover and Mitra, 2017; Gregoire and Langlais, 2018) learn to identify parallel sentences in the semantic spaces. However, these methods require large amounts of parallel sentence pairs to train the systems first and then test the performance on raw comparable corpora, which does not apply to languages with limited resources. Instead, we explore the corpora mining in an unsupervised fashion and propose a joint training framework with machine translation. Retrieval-Augmented Text Generation Our work is also related to the recent work on applying retrieval mechanisms to augment text generation, such as image captioning (Kuznetsova et al., 2013; Mason and Charniak, 2014), dialogue generation (Song et al., 2016; Yan et al., 2016; Wu et al., 2018) and style transfer (Lin et al., 2017; Li et al., 2018). Some editing-based models (Guu et al., 2018; Hashimoto et al., 2018) are proposed to further enhance the retrieved text. Recent work in machine translation (Gu et al., 2018) augments an NMT model with sentence pairs retrieved by an off-the-shelf search engine. However, these methods are two-staged with supervised retrieval first. In our work, the extracted-edited sentences are not directly used as the ground truth to train the translation model. Instead, we view these sentences as pivotal points in the target language space and further we propose a comparative translation loss to train the system in a fully unsupervised way. In this paper, we propose an extract-edit approach, an effective alternative to the widely-used back-translation in unsupervised NMT. Instead of generating pseudo language pairs to train the systems with the reconstruction loss, we design a comparative translation loss that leverages real sentences in the target language space. Empirically, our method advances the previous state-of-the-art NMT systems across four language pairs using the monolingual corpora only. Theoretically, we believe the extract-edit learning framework can be generalized to other types of unsupervised machine translation systems and even some other unsupervised learning tasks. The authors would like to thank the anonymous reviewers for their thoughtful comments. The work was supported by the Facebook Low Resource Neural Machine Translation Research Award. The authors are solely responsible for the contents of the paper, and the opinions expressed in this publication do not reflect those of the funding agencies.
abstain abstain abstain objective method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain objective objective objective abstain abstain abstain abstain method abstain other other other other method abstain other other abstain other other objective other method abstain abstain other abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain other abstain abstain abstain method abstain method abstain method abstain abstain abstain abstain abstain abstain method abstain abstain other method abstain method abstain other method abstain abstain abstain abstain method abstain abstain other abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain other abstain abstain abstain result result abstain abstain result abstain abstain abstain abstain result result abstain result abstain result abstain abstain result abstain abstain abstain abstain result abstain result result result abstain abstain abstain abstain result abstain abstain result abstain abstain abstain result other other other other abstain other other other other objective other other other other method objective objective method method method other other other