Published as a conference paper at ICLR 2018 UNSUPERVISED NEURAL MACHINE TRANSLATION
In spite of the recent success of neural machine translation (NMT) in standard benchmarks, the lack of large parallel corpora poses a major practical problem for many language pairs. There have been several proposals to alleviate this issue with, for instance, triangulation and semi-supervised learning techniques, but they still require a strong cross-lingual signal. In this work, we completely remove the need of parallel data and propose a novel method to train an NMT system in a completely unsupervised manner, relying on nothing but monolingual corpora. Our model builds upon the recent work on unsupervised embedding mappings, and consists of a slightly modified attentional encoder-decoder model that can be trained on monolingual corpora alone using a combination of denoising and backtranslation. Despite the simplicity of the approach, our system obtains 15.56 and 10.21 BLEU points in WMT 2014 French → English and German → English translation. The model can also profit from small parallel corpora, and attains 21.81 and 15.24 points when combined with 100,000 parallel sentences, respectively. Our implementation is released as an open source project 1 .
unsupervised neural machine translation
This is an interesting paper on unsupervised MT. It trains a standard architecture using:
1) word embeddings in a shared embedding space, learned using a recent approach that works with only tens of bilingual word papers.
2) A encoder-decoder trained using only monolingual data (should cite http://www.statmt.org/wmt17/pdf/WMT15.pdf). Training uses a “denoising” method which is not new: it uses the same idea as contrastive estimation (http://www.aclweb.org/anthology/P05-1044, a well-known method which should be cited). 
3) Backtranslation.
All though none of these ideas are new, they haven’t been combined in this way before, and that what’s novel here. The paper is essentially a neat application of (1), and is an empirical/ systems paper. It’s essentially a proof-of-concept that it is that it’s possible to get anything at all using no parallel data. That’s surprising and interesting, but I learned very little else from it. The paper reads as preliminary and rushed, and I had difficulty answering some basic questions:
* In Table (1), I’m slightly puzzled by why 5 is better than 6, and this may be because I’m confused about what 6 represents. It would be natural to compare 5 with a system trained on 100K parallel text, since the systems would then (effectively) differ only in that 5 also exploits additional monolingual data. But the text suggests that 6 is trained on much more than 100K parallel sentences; that is, it differs in at least two conditions (amount of parallel text and use of monolingual text). Since this paper’s primary contribution is empirical, this comparison should be done in a carefully controlled way, differing each of these elements in turn.
* I’m very confused by the comment on p. 8 that “the modifications introduced by our proposal are also limiting” to the “comparable supervised NMT system”. According to the paper, the architecture of the system is unchanged, so why would this be the case? This comment makes it seem like something else has been changed in the baseline, which in turn makes it somewhat hard to accept the results here.
Comment:
* The qualitative analysis is not really an analysis: it’s just a few cherry-picked examples and some vague observations. While it is useful to see that the system does indeed generate nontrivial content in these cases, this doesn’t give us further insight into what the system does well or poorly outside these examples. The BLEU scores suggest that it also produces many low-quality translations. What is different about these particular examples? (Aside: since the cross-lingual embedding method is trained on numerals, should we be concerned that the system fails at translating numerals?)
Questions:
* Contrastive estimation considers other neighborhood functions (“random noise” in the parlance of this paper), and it’s natural to wonder what would happen if this paper also used these or other neighborhood functions. More importantly, I suspect the the neighborhood functions are important: when translating between Indo-European languages as in these experiments, local swaps are reasonable; but in translating between two different language families (as would often be the case in the motivating low-resource scenario that the paper does not actually test), it seems likely that other neighborhood functions would be important, since structural differences would be much larger.
Presentational comments (these don’t affect my evaluation, they’re mostly observations but they contribute to a general feeling that the paper is rushed and preliminary):
* BPE does not “learn”, it’s entirely deterministic.
* This paper is at best tangentially related to decipherment. Decipherment operates under two quite different assumptions: there is no training data for the source language ciphertext, only the ciphertext itself (which is often very small); and the replacement function is deterministic rather than probabilistic (and often monotonic). The Dou and Knight papers are interesting, but they’re an adaptation of ideas rather than decipherment per se. Since none of those ideas are used here this feels like hand-waving.
* Future work is vague: “we would like to detect and mitigate the specific causes…” “we also think that a better handling of rare words…” That’s great, but how will you do these things? Do you have specific reasons to think this, or ideas on how to approach them? Otherwise this is just hand-waving.