Pascanu, R., Mikolov, T., and Bengio, Y . (2013b). On the difﬁculty of training recurrent neural
networks. In Proceedings of the 30th International Conference on Machine Learning (ICML
2013) .
Pascanu, R., Gulcehre, C., Cho, K., and Bengio, Y . (2014). How to construct deep recurrent neural
networks. In Proceedings of the Second International Conference on Learning Representations
(ICLR 2014) .
Pouget-Abadie, J., Bahdanau, D., van Merri ¨enboer, B., Cho, K., and Bengio, Y . (2014). Overcoming
the curse of sentence length for neural machine translation using automatic segmentation. In
Eighth Workshop on Syntax, Semantics and Structure in Statistical Translation . to appear.
Schuster, M. and Paliwal, K. K. (1997). Bidirectional recurrent neural networks. Signal Processing,
IEEE Transactions on ,45(11), 2673–2681.
Schwenk, H. (2012). Continuous space translation models for phrase-based statistical machine
translation. In M. Kay and C. Boitet, editors, Proceedings of the 24th International Conference on
Neural machine translation is a recently proposed approach to machine
translation. Unlike the traditional statistical machine translation, the neural
machine translation aims at building a single neural network that can be
jointly tuned to maximize the translation performance. The models proposed
recently for neural machine translation often belong to a family of
encoder-decoders and consists of an encoder that encodes a source sentence into
a fixed-length vector from which a decoder generates a translation. In this
paper, we conjecture that the use of a fixed-length vector is a bottleneck in
improving the performance of this basic encoder-decoder architecture, and
propose to extend this by allowing a model to automatically (soft-)search for
parts of a source sentence that are relevant to predicting a target word,
without having to form these parts as a hard segment explicitly. With this new
approach, we achieve a translation performance comparable to the existing
state-of-the-art phrase-based system on the task of English-to-French
translation. Furthermore, qualitative analysis reveals that the
(soft-)alignments found by the model agree well with our intuition.
Dzmitry Bahdanau Kyunghyun Cho Yoshua Bengio