Dual Learning for Machine Translation
While neural machine translation (NMT) is making good progress in the past two years, tens of millions of bilingual sentence pairs are needed for its training. However, human labeling is very costly. To tackle this training data bottleneck, we develop a dual-learning mechanism, which can enable an NMT system to automatically learn from unlabeled data through a dual-learning game. This mechanism is inspired by the following observation: any machine translation task has a dual task, e.g., English-to-French translation (primal) versus French-to-English translation (dual); the primal and dual tasks can form a closed loop, and generate informative feedback signals to train the translation models, even if without the involvement of a human labeler. In the dual-learning mechanism, we use one agent to represent the model for the primal task and the other agent to represent the model for the dual task, then ask them to teach each other through a reinforcement learning process. Based on the feedback signals generated during this process (e.g., the languagemodel likelihood of the output of a model, and the reconstruction error of the original sentence after the primal and dual translations), we can iteratively update the two models until convergence (e.g., using the policy gradient methods). We call the corresponding approach to neural machine translation dual-NMT. Experiments show that dual-NMT works very well on English↔French translation; especially, by learning from monolingual data (with 10% bilingual data for warm start), it achieves a comparable accuracy to NMT trained from the full bilingual data for the French-to-English translation task.
This paper presents a model called "communication-based machine translation (CMT), where translation models for the two translation directions are improved jointly by forth- and back-translating monolingual data and collecting feedback after each sampled translation. The goal is to leverage monolingual data in an effective way for improving machine translation based on bilingual corpora. The model is evaluated on two language pairs and compared to a standard neural MT system and a system that is additionally trained on pseudo-bilingual data.
The same goal has been pursued by e.g. including a language model on the target side for re-scoring translations [1] or include it into the MT system via deep or shallow fusion [2]. The paper does not sufficiently review the work that has been done in this direction and only focuses on the recent work by Sennrich et al. Since the goal of exploiting monolingual data for MT has been in the focus of many works, more empirical comparisons are needed to demonstrate the superiority of their system. It would have been easy to e.g. use the same data set as in [2] to make a direct comparison. Also, there has been work on the unsupervised training of noisy-channel models [3] which needs to be mentioned. In addition, there are no comparisons to other communication-based learning scenarios in reinforcement learning. Instead of repeatingly describing the basic idea of having two players communicating via two translation models, the paper would benefit from a more careful and thorough review and comparisons of previous work. The authors conclude that this model gives new directions for learning the translation model from scratch from monolingual data, but I personally doubt that this will work, since both Ranzato et. al. and Shen et al. [4] who learn translation models from reinforcement learning observed that a warm-start model is required, because the feedback is too weak. In line 116 they assume that translation starts with "weak" models, but in the end they use "well-trained" (l 152) translation models. The only experiment which suppports that the improvement is really large with monolingual data and a "weak" model is Fr->En.  Also the extension to multiple languages in a translation chain is not as straight-forward as described, because the source of translation errors cannot be detected that easily anymore, so the updates might get very noisy. Although the metaphor of the two-players-game is nice, it does not align well with the actual algorithm. The assumption is that each player only understands one language, so scoring via a language model is possible, but the communication reward requires knowledge of the actual translation model, which is unavailable for the player in the described game. In order to accept the experiments as supporting their model, more details about the models (number of training instances - monolingual and bilingual) need to be reported and better comparisons have to be made (on the original data from the referred work and state-of-the-art as upper bound). To put it in a nutshell, they present a nice (but not so novel) idea, but the paper lacks sufficient (experimental) comparisons to closely related work for me to accept it.  [1] Holger Schwenk. 2012. Continuous space translation models for phrase-based statistical machine translation. In Martin Kay and Christian Boitet, editors, Proceedings of the 24th International Conference on Computational Linguistics (COLIN), pages 1071–1080. Indian Institute of Technology Bombay. [2] Gulcehre, C., Firat, O., Xu, K., Cho, K., Barrault, L., Lin, H. C., ... & Bengio, Y. (2015). On using monolingual corpora in neural machine translation. arXiv preprint arXiv:1503.03535. [3] Mylonakis, Markos, Khalil Sima'an, and Rebecca Hwa. "Unsupervised estimation for noisy-channel models." Proceedings of the 24th international conference on Machine learning. ACM, 2007. [4] Shen, S., Cheng, Y., He, Z., He, W., Wu, H., Sun, M., & Liu, Y. (2015). Minimum risk training for neural machine translation. arXiv preprint arXiv:1512.02433.