Training neural machine translation models (NMT) requires a large amount of parallel corpus, which is scarce for many language pairs. However, raw non-parallel corpora are often easy to obtain. Existing approaches have not exploited the full potential of non-parallel bilingual data either in training or decoding. In this paper, we propose the mirror-generative NMT (MGNMT), a single unified architecture that simultaneously integrates the source to target translation model, the target to source translation model, and two language models. Both translation models and language models share the same latent semantic space, therefore both translation directions can learn from non-parallel data more effectively. Besides, the translation models and language models can collaborate together during decoding. Our experiments show that the proposed MGNMT consistently outperforms existing approaches in all a variety of scenarios and language pairs, including resourcerich and low-resource languages.
This paper proposes an approach to neural MT in which the joint (source, target) distribution is modeled as an average over two different factorizations: target given source and source given target. This gives rise to four distributions - two language models and two translation models - which are parameterized separately but conditioned on a common variational latent variable.  The model is trained on parallel data using a standard VAE approach. It can additionally be trained on non-parallel data in an approach similar to iterated back-translation at sentence-level granularity, but with language and translation model probabilities for observed sentences coupled by the latent variable. Inference iterates between sampling a latent variable given the current best hypothesis, and using beam search plus rescoring to find a new best hypothesis given the current latent variable. The approach is evaluated in several different scenarios (low- and high-resource, domain adaptation - trained on parallel data only or parallel plus monolingual data) and found to generally outperform previous work on generative NMT and iterated back-translation.
Strengths: clearly written, well motivated, very comprehensive experiments comparing to relevant baselines.
Weaknesses: somewhat incremental relative to Shah and Barber (Neurips 2018), results are only marginally positive, framework is probably too cumbersome to justify widespread adoption based on the results.
I think the paper should be accepted. Although it’s not highly original, it ties together three strands of work in a principled way: joint models, variational approaches, and back-translation / dual learning. The increment over Shah and Barber is bolstered by the addition of back-translation, which gives substantial improvements when using non-parallel data; and to a lesser extent by the argument about the advantage of separate models for distant language pairs. Using all possible LMs and TMs coupled with a latent variable feels like an area that was inevitably going to get explored, and this paper does a good job of it. Although the gains over the baselines are not overly compelling, they are quite systematic, indicating that the advantage is probably real, albeit slight. The authors are also to be commended on their use of not just the Shah and Barber baseline, but also the back-translation-based techniques, which are generally stronger competitors when monolingual data is incorporated.
Further comments/questions:
Why are there no results for Transformer+Dual in table 4? This omission looks odd, since Transformer+Dual was the strongest baseline in table 3.
Please add implementation details for the Transformer+{BT,JBT,Dual} baselines.
It was surprising not to see robustness experiments like Shah and Barber’s dropped source words, since robustness to source noise could be one of the advantages of having an explicit model of the source sentence.
A few additional suggestions for related work: noisy channel approaches (eg, The Neural Noisy Channel, Yu et al, ICLR 2017); decipherment (eg, Beyond parallel data: Joint word alignment and decipherment improves machine translation, EMNLP 2014 - yes, from SMT days, but still); other joint modeling work (KERMIT: Generative Insertion-Based Modeling for Sequences, Chan et al, 2019).
Consider dropping the “1+1 > 2” metaphor. It’s not clear to me exactly what it means, or what it adds to the paper.
“deviation” is used in a couple places where you probably meant “derivation”?
Line 6 in algorithm 2 should use both forward and backward scores for rescoring.