Unsupervised Machine Translation Using Monolingual Corpora Only
Machine translation has recently achieved impressive performance thanks to recent advances in deep learning and the availability of large-scale parallel corpora. There have been numerous attempts to extend these successes to low-resource language pairs, yet requiring tens of thousands of parallel sentences. In this work, we take this research direction to the extreme and investigate whether it is possible to learn to translate even without any parallel data. We propose a model that takes sentences from monolingual corpora in two different languages and maps them into the same latent space. By learning to reconstruct in both languages from this shared feature space, the model effectively learns to translate without using any labeled data. We demonstrate our model on two widely used datasets and two language pairs, reporting BLEU scores of 32.8 and 15.1 on the Multi30k and WMT English-French datasets, without using even a single parallel sentence at training time.
 This paper describes an approach to train a neural machine translation system without parallel data. Starting from a word-to-word translation lexicon, which was also learned with unsupervised methods, this approach combines a denoising auto-encoder objective with a back-translation objective, both in two translation directions, with an adversarial objective that attempts to fool a discriminator that detects the source language of an encoded sentence. These five objectives together are sufficient to achieve impressive English <-> German and Engish <-> French results in Multi30k, a bilingual image caption scenario with short simple sentences, and to achieve a strong start for a standard WMT scenario. This is very nice work, and I have very little to criticize. The approach is both technically interesting, and thorough in that it explores and combines a host of ideas that could work in this space (initial bilingual embeddings, back translation, auto-encoding, and adversarial techniques). And it is genuinely impressive to see all these pieces come together into something that translates substantially better than a word-to-word baseline. But the aspect I like most about this paper is the experimental analysis. Considering that this is a big, complicated system, it is crucial that the authors included both an ablation experiment to see which pieces were most important, and an experiment that indicates the amount of labeled data that would be required to achieve the same results with a supervised system. In terms of specific criticisms: In Equations (2), consider replacing C(y) with C(M(x)), or use compose notation, in order to make x-hat*s relationship to x clear and self-contained within the equation. I am glad you take the time to give your model selection criterion it*s own section in 3.2, as it does seem to be an important part of this puzzle. However, it would be nice to provide actual correlation statistics rather than an anecdotal illustration of correlation. In the first paragraph of Section 4.5, I disagree with the sentence, "Similar observations can be made for the other language pairs we considered." In fact, I would go so far as to say that the English to French scenario described in that paragraph is a notable outlier, in that it is the other language pair where you beat the oracle re-ordering baseline in both Multi30k and WMT. When citing Shen et al., 2017, consider also mentioning the following: Controllable Invariance through Adversarial Feature Learning; Qizhe Xie, Zihang Dai, Yulun Du, Eduard Hovy, Graham Neubig; NIPS 2017; https://arxiv.org/abs/1705.11122 Response read -- thanks.
You are a member of the scientific community tasked with peer review. Review the following paper title and abstract.
### Title + Abstract
Unsupervised Machine Translation Using Monolingual Corpora Only
Machine translation has recently achieved impressive performance thanks to recent advances in deep learning and the availability of large-scale parallel corpora. There have been numerous attempts to extend these successes to low-resource language pairs, yet requiring tens of thousands of parallel sentences. In this work, we take this research direction to the extreme and investigate whether it is possible to learn to translate even without any parallel data. We propose a model that takes sentences from monolingual corpora in two different languages and maps them into the same latent space. By learning to reconstruct in both languages from this shared feature space, the model effectively learns to translate without using any labeled data. We demonstrate our model on two widely used datasets and two language pairs, reporting BLEU scores of 32.8 and 15.1 on the Multi30k and WMT English-French datasets, without using even a single parallel sentence at training time.
user
<think>Okay, to do a decent peer review, I should start with summarizing what the work is about. This includes novelty, contributions, clarity, and impact. I might include suggestions to issues that I find critical to the usefulness of the work. Starting with novelty, there are different types of novelty. The first aspect of novelty is scope and scale. Is the work covering a broader perspective than previous works? The second aspect is methodology. Is the proposed methodology novel compared to previous methods? Do they provide any significant advantage over previous methodologies? The third aspect is results. Are the results different compared previous work? Does this lead to any novel theories or an extension of current theories? Next is clarity, I will be reading the work in depth and analyze if any sentences can be potentially ambiguous or misleading. If any sentences are ambiguous, I will state them and give a suggestion on how to rephrase such sentence or how to make more clear to the audience. Lastly, impact is important. I will attempt to predict how such work will impact the scientific community. This might include the applications of such work and how the work might transform the way we study a certain topic. Finally, I will might only mention issues that are critical. This is to keep the review as concise and helpful as possible.</think>



 This paper describes an approach to train a neural machine translation system without parallel data. Starting from a word-to-word translation lexicon, which was also learned with unsupervised methods, this approach combines a denoising auto-encoder objective with a back-translation objective, both in two translation directions, with an adversarial objective that attempts to fool a discriminator that detects the source language of an encoded sentence. These five objectives together are sufficient to achieve impressive English <-> German and Engish <-> French results in Multi30k, a bilingual image caption scenario with short simple sentences, and to achieve a strong start for a standard WMT scenario. This is very nice work, and I have very little to criticize. The approach is both technically interesting, and thorough in that it explores and combines a host of ideas that could work in this space (initial bilingual embeddings, back translation, auto-encoding, and adversarial techniques). And it is genuinely impressive to see all these pieces come together into something that translates substantially better than a word-to-word baseline. But the aspect I like most about this paper is the experimental analysis. Considering that this is a big, complicated system, it is crucial that the authors included both an ablation experiment to see which pieces were most important, and an experiment that indicates the amount of labeled data that would be required to achieve the same results with a supervised system. In terms of specific criticisms: In Equations (2), consider replacing C(y) with C(M(x)), or use compose notation, in order to make x-hat*s relationship to x clear and self-contained within the equation. I am glad you take the time to give your model selection criterion it*s own section in 3.2, as it does seem to be an important part of this puzzle. However, it would be nice to provide actual correlation statistics rather than an anecdotal illustration of correlation. In the first paragraph of Section 4.5, I disagree with the sentence, "Similar observations can be made for the other language pairs we considered." In fact, I would go so far as to say that the English to French scenario described in that paragraph is a notable outlier, in that it is the other language pair where you beat the oracle re-ordering baseline in both Multi30k and WMT. When citing Shen et al., 2017, consider also mentioning the following: Controllable Invariance through Adversarial Feature Learning; Qizhe Xie, Zihang Dai, Yulun Du, Eduard Hovy, Graham Neubig; NIPS 2017; https://arxiv.org/abs/1705.11122 Response read -- thanks.
assistant
 This paper describes an approach to train a neural machine translation system without parallel data. Starting from a word-to-word translation lexicon, which was also learned with unsupervised methods, this approach combines a denoising auto-encoder objective with a back-translation objective, both in two translation directions, with an adversarial objective that attempts to fool a discriminator that detects the source language of an encoded sentence. These five objectives together are sufficient to achieve impressive English <-> German and Engish <-> French results in Multi30k, a bilingual image caption scenario with short simple sentences, and to achieve a strong start for a standard WMT scenario. This is very nice work, and I have very little to criticize. The approach is both technically interesting, and thorough in that it explores and combines a host of ideas that could work in this space (initial bilingual embeddings, back translation, auto-encoding, and adversarial techniques). And it is genuinely impressive to see all these pieces come together into something that translates substantially better than a word-to-word baseline. But the aspect I like most about this paper is the experimental analysis. Considering that this is a big, complicated system, it is crucial that the authors included both an ablation experiment to see which pieces were most important, and an experiment that indicates the amount of labeled data that would be required to achieve the same results with a supervised system. In terms of specific criticisms: In Equations (2), consider replacing C(y) with C(M(x)), or use compose notation, in order to make x-hat*s relationship to x clear and self-contained within the equation. I am glad you take the time to give your model selection criterion it*s own section in 3.2, as it does seem to be an important part of this puzzle. However, it would be nice to provide actual correlation statistics rather than an anecdotal illustration of correlation. In the first paragraph of Section 4.5, I disagree with the sentence, "Similar observations can be made for the other language pairs we considered." In fact, I would go so far as to say that the English to French scenario described in that paragraph is a notable outlier, in that it is the other language pair where you beat the oracle re-ordering baseline in both Multi30k and WMT. When citing Shen et al., 2017, consider also mentioning the following: Controllable Invariance through Adversarial Feature Learning; Qizhe Xie, Zihang Dai, Yulun Du, Eduard Hovy, Graham Neubig; NIPS 2017; https://arxiv.org/abs/1705.11122 Response read -- thanks.