Published as a conference paper at ICLR 2018 TOWARDS NEURAL PHRASE-BASED MACHINE TRANSLATION
In this paper, we present Neural Phrase-based Machine Translation (NPMT).
1 Our method explicitly models the phrase structures in output sequences using SleepWAke Networks (SWAN), a recently proposed segmentation-based sequence modeling method. To mitigate the monotonic alignment requirement of SWAN, we introduce a new layer to perform (soft) local reordering of input sequences. Different from existing neural machine translation (NMT) approaches, NPMT does not use attention-based decoding mechanisms. Instead, it directly outputs phrases in a sequential order and can decode in linear time. Our experiments show that NPMT achieves superior performances on IWSLT 2014 German-English/EnglishGerman and IWSLT 2015 English-Vietnamese machine translation tasks compared with strong NMT baselines. We also observe that our method produces meaningful phrases in output languages.
Authors proposed a new neural-network based machine translation method that generates the target sentence by generating multiple partial segments in the target sentence from different positions in the source information. The model is based on the SWAN architecture which is previously proposed, and an additional "local reordering" layer to reshuffle source information to adjust those positions to the target sentence.
Using the SWAN architecture looks more reasonable than the conventional attention mechanism when the ground-truth word alignment is monotone. Also, the concept of local reordering mechanism looks well to improve the basic SWAN model to reconfigure it to the situation of machine translation tasks.
The "window size" of the local reordering layer looks like the "distortion limit" used in traditional phrase-based statistical machine translation methods, and this hyperparameter may impose a similar issue with that of the distortion limit into the proposed model; small window sizes may drop information about long dependency. For example, verbs in German sentences sometimes move to the tail of the sentence and they introduce a dependency between some distant words in the sentence. Since reordering windows restrict the context of each position to a limited number of neighbors, it may not capture distant information enough. I expected that some observations about this point will be unveiled in the paper, but unfortunately, the paper described only a few BLEU scores with different window sizes which have not enough information about it. It is useful for all followers of this paper to provide some observations about this point.
In addition, it could be very meaningful to provide some experimental results on linguistically distant language pairs, such as Japanese and English, or simply reversing word orders in either source or target sentences (this might work to simulate the case of distant reordering).
Authors argued some differences between conventional attention mechanism and the local reordering mechanism, but it is somewhat unclear that which ones are the definite difference between those approaches.
A super interesting and mysterious point of the proposed method is that it achieves better BLEU than conventional methods despite no any global language models (Table 1 row 8), and the language model options (Table 1 row 9 and footnote 4) may reduce the model accuracy as well as it works not so effectively. This phenomenon definitely goes against the intuitions about developing most of the conventional machine translation models. Specifically, it is unclear how the model correctly treats word connections between segments without any global language model. Authors should pay attention to explain more detailed analysis about this point in the paper.
Eq. (1) is incorrect. According to Fig. 2, the conditional probability in the product operator should be revised to p(a_t | x_{1:t}, a_{1:t-1}), and the independence approximation to remove a_{1:t-1} from the conditions should also be noted in the paper.
Nevertheless, the condition x_{1:t} could not be reduced because the source position is always conditioned by all previous positions through an RNN.