  Bilingual sequence models improve phrase-based translation and reordering by
overcoming phrasal independence assumption and handling long range reordering.
However, due to data sparsity, these models often fall back to very small
context sizes. This problem has been previously addressed by learning sequences
over generalized representations such as POS tags or word clusters. In this
paper, we explore an alternative based on neural network models. More
concretely we train neuralized versions of lexicalized reordering and the
operation sequence models using feed-forward neural network. Our results show
improvements of up to 0.6 and 0.5 BLEU points on top of the baseline
German->English and English->German systems. We also observed improvements
compared to the systems that used POS tags and word clusters to train these
models. Because we modify the bilingual corpus to integrate reordering
operations, this allows us to also train a sequence-to-sequence neural MT model
having explicit reordering triggers. Our motivation was to directly enable
reordering information in the encoder-decoder framework, which otherwise relies
solely on the attention model to handle long range reordering. We tried both
coarser and fine-grained reordering operations. However, these experiments did
not yield any improvements over the baseline Neural MT systems.

Computation and Language