Generative Neural Machine Translation
We introduce Generative Neural Machine Translation (GNMT), a latent variable architecture which is designed to model the semantics of the source and target sentences. We modify an encoder-decoder translation model by adding a latent variable as a language agnostic representation which is encouraged to learn the meaning of the sentence. GNMT achieves competitive BLEU scores on pure translation tasks, and is superior when there are missing words in the source sentence. We augment the model to facilitate multilingual translation and semi-supervised learning without adding parameters. This framework significantly reduces overfitting when there is limited paired data available, and is effective for translating between pairs of languages not seen during training.
Summary  This paper proposes a generative latent variable model for neural machine translation, where inference is performed with variational inference. This extends the work of Zhang et al., 2016, who proposed a conditional model with variational inference. The advantage of the generative model is to force the latent variable to capture more of the semantics of the sentence than the conditional model was able to do. The main disadvantage of this approach is that the value of the latent variable has to be infered during decoding (based on candidate generations).   The paper also shows that a version of this model can be trained in a multilingual setting, that monolingual data can be used as semi-supervised training, and that the inference algorithm can be extended to perform translation with missing words.      Results show that the model generates better translations than the Variational NMT baseline across multiple language pairs and training data sizes, although the improvement is larger in a small data setting. Multilingual training and semi-supervised training also gives substantial improvements in small data settings.   Quality  An alternative version of this model would be to condition the inference network q only on x, not on x and y. This would make inference during decoding more efficient, and as z should encode the semantics of either x or y, it is not clear what the additional benefit is of conditioning on both.   Experiments: why not also compare directly with previous results on more standard datasets, especially with Zhang et al. 2016's reported results? The paper is also missing a comparison with a standard NMT model without latent variables, without which its performance cannot fully be assessed.   Originality  The extension over Zhang et al., 2016 is relatively small in terms of the technical contribution, but I do think that the generative formulation has clear advantages over previous work.  Significance  The main advantage of this approach seems to be in better sampling complexity (more data-efficient learning) than baselines, which is an important direction in machine translation research.     More analysis of what semantics the latent vector is learning to capture would be interesting - for example by sampling x and y given z inference on a observed sentence or sentence pair.   Overall I think this is an interesting contribution that I'd like to see accepted.