Hochreiter, S. and Schmidhuber, J. (1997). Long short-term memory. Neural Computation ,9(8),
1735–1780.
Kalchbrenner, N. and Blunsom, P. (2013). Recurrent continuous translation models. In Proceedings
of the ACL Conference on Empirical Methods in Natural Language Processing (EMNLP) , pages
1700–1709. Association for Computational Linguistics.
Koehn, P. (2010). Statistical Machine Translation . Cambridge University Press, New York, NY ,
USA.
Koehn, P., Och, F. J., and Marcu, D. (2003). Statistical phrase-based translation. In Proceedings
of the 2003 Conference of the North American Chapter of the Association for Computational
Linguistics on Human Language Technology - Volume 1 , NAACL ’03, pages 48–54, Stroudsburg,
PA, USA. Association for Computational Linguistics.
Pascanu, R., Mikolov, T., and Bengio, Y . (2013a). On the difﬁculty of training recurrent neural
networks. In ICML’2013 .
Pascanu, R., Mikolov, T., and Bengio, Y . (2013b). On the difﬁculty of training recurrent neural
Neural machine translation is a recently proposed approach to machine
translation. Unlike the traditional statistical machine translation, the neural
machine translation aims at building a single neural network that can be
jointly tuned to maximize the translation performance. The models proposed
recently for neural machine translation often belong to a family of
encoder-decoders and consists of an encoder that encodes a source sentence into
a fixed-length vector from which a decoder generates a translation. In this
paper, we conjecture that the use of a fixed-length vector is a bottleneck in
improving the performance of this basic encoder-decoder architecture, and
propose to extend this by allowing a model to automatically (soft-)search for
parts of a source sentence that are relevant to predicting a target word,
without having to form these parts as a hard segment explicitly. With this new
approach, we achieve a translation performance comparable to the existing
state-of-the-art phrase-based system on the task of English-to-French
translation. Furthermore, qualitative analysis reveals that the
(soft-)alignments found by the model agree well with our intuition.
Dzmitry Bahdanau Kyunghyun Cho Yoshua Bengio