We examine how learning from unaligned data can improve both the data efficiency of supervised tasks as well as enable alignments without any supervision. For example, consider unsupervised machine translation: the input is two corpora of English and French, and the task is to translate from one language to the other but without any pairs of English and French sentences. To address this, we develop feature matching auto-encoders (FMAEs). FMAEs ensure that the marginal distribution of feature layers are preserved across forward and inverse mappings between domains. We show that FMAEs achieve state of the art for data efficiency and alignment across three tasks: text decipherment, sentiment transfer, and neural machine translation for English-to-German and English-to-French. Most compellingly, FMAEs achieve state of the art for semi-supervised neural machine translation with significant BLEU score differences of up to 5.7 and 6.3 over traditional supervised models. Furthermore, on English-to-German, FMAEs outperform last year's best models such as ByteNet (Kalchbrenner et al., 2016) while using only half as many supervised examples.
This paper proposes a generative model called matching auto-encoder to carry out the learning from unaligned data.
However, it is very disappointed to read the contents after the introduction, since most of the contributions are overclaimed.
Detailed comments:
- Figure 1 is incorrect because the pairs (x, z) and (y, z) should be put into two different plates if  x and y are unaligned.
- Lots of contents in Sec. 3 are confusing to me. What is the difference between g_l(x) and g_l(y) if g_l : H_{l−1} → H_l and f_l: H_{l−1} → H_l are the same? What are e_x and e_y? Why is there a λ if it is a generative model?
- If the title is called 'text decipherment', there should be no parallel data at all, otherwise it is a huge overclaim on the decipherment tasks. Please add citations of Kevin Knight's recent papers on deciperment.
- Reading the experiment results of 'Sentiment Transfer' is a disaster to me. I couldn't get much information on 'sentiment transfer' from a bunch of ungrammatical unnatural language sentences. I would prefer to see some results of baseline models for comparison instead of a pure qualitative analysis.
- The claim on "FMAEs are state of the art for neural machine translation with limited supervision on EN-DE and EN-FR" is not exciting to me. Semi-supervised learning is interesting, but in the scenario of MT we do have enough parallel data for many language pairs. Unless the model is able to exceed the 'real' state-of-the-art that uses the full set of parallel data, otherwise we couldn't identify whether the models are able to benefit NMT.  Interestingly, the authors didn't provide any of the results that are experimented with full parallel data set. Possibly it is because the introduction of stochastic variables that prevent the models from overfitting on small datasets.