Planning is important for humans when producing complex languages, which is a missing part in current language generation models. In this work, we add a planning phase in neural machine translation to control the global sentence structure ahead of translation. Our approach learns discrete structural representations to encode syntactic information of target sentences. During translation, we can either let beam search to choose the structural codes automatically or specify the codes manually. The word generation is then conditioned on the selected discrete codes. Experiments show that the translation performance remains intact by learning the codes to capture pure structural variations. Through structural planning, we are able to control the global sentence structure by manipulating the codes. By evaluating with a proposed structural diversity metric, we found that the sentences sampled using different codes have much higher diversity scores. In qualitative analysis, we demonstrate that the sampled paraphrase translations have drastically different structures.
The authors propose modeling structural diversity of translations by conditioning the generation on both the source sentence and a latent encoding of the overall structure (captured by simplified part-of-speech tags). Specifically, they first train a conditional autoencoder to learn a latent code optimized towards reconstructing the tag sequence. They then prefix the inferred latent code to the target sentence before generation. A diversity metric which measures pairwise BLEU scores between beam items is also proposed. Experiments show that the latent codes lead to greater structural diversity as well as marginally improved translation results when combined with beam search.
Contributions
-----------------
A simple method for improving structural diversity.
The use of conditional autoencoding to capture structural ambiguity, while not in itself novel, could be interesting for other problems as well.
Experiments suggest that the method is rather effective (albeit only improving translation quality marginally)
I like the proposed discrepancy score based on pairwise BLEU scores.
Issues
---------
It is not clear if teacher forcing was used in the "tag planning" setting. If gold tag sequences were used during training there is a major train/test mismatch which would explain the dramatic drop in BLEU scores. If so, this is a major issue, since the authors claim that as the motivation for the use of discrete latent codes. To make the "tag planning" setting comparable to the latent code setting, you would need to train the tag prediction model first and then condition on predicted tags when training the translation model (potentially you would need to do jack-knifing to prevent overfitting as well).
It is unfortunate that there is no empirical comparison with the most closely related prior work, in particular Li et al. (2016) and Xu et al. (2018), which are both appropriately cited. As it stands it is not possible to tell which of these approaches is most useful in practice.
No details are provided on the tagset used and what system is used to predict it, or to what degree of accuracy.
Having a fixed number of codes regardless of sentence length seems like a major shortcoming. I would urge the authors to consider a variable coding length scheme, e.g., by generating codes autoregressively instead of with a fixed number of softmaxes. It would also be interesting to break down the numbers in table 1 with respect to sentence length.
Minor issues
-----------------
Citation for the Xavier method is missing.
Notation is somewhat hard to follow. Please add a few sentences describing it and make sure it is consistent.
There are many grammatical errors. Please make sure to proofread!
"Please note that the planning component can also be a continuous latent vector, which requires a discriminator to train the model in order that the latent cap." What does this mean?