decodes to a target sentence y. For instance, two recurrent neural networks (RNN) were used by
(Cho et al. , 2014a) and (Sutskever et al. , 2014) to encode a variable-length source sentence into a
ﬁxed-length vector and to decode the vector into a variable-length target sentence.
Despite being a quite new approach, neural machine translation has already shown promising results.
Sutskever et al. (2014) reported that the neural machine translation based on RNNs with long shortterm memory (LSTM) units achieves close to the state-of-the-art performance of the conventional
phrase-based machine translation system on an English-to-French translation task.1Adding neural
components to existing translation systems, for instance, to score the phrase pairs in the phrase
table (Cho et al. , 2014a) or to re-rank candidate translations (Sutskever et al. , 2014), has allowed to
surpass the previous state-of-the-art performance level.
2.1 RNN E NCODER –DECODER
Here, we describe brieﬂy the underlying framework, called RNN Encoder–Decoder , proposed by
Cho et al. (2014a) and Sutskever et al. (2014) upon which we build a novel architecture that learns
Neural machine translation is a recently proposed approach to machine
translation. Unlike the traditional statistical machine translation, the neural
machine translation aims at building a single neural network that can be
jointly tuned to maximize the translation performance. The models proposed
recently for neural machine translation often belong to a family of
encoder-decoders and consists of an encoder that encodes a source sentence into
a fixed-length vector from which a decoder generates a translation. In this
paper, we conjecture that the use of a fixed-length vector is a bottleneck in
improving the performance of this basic encoder-decoder architecture, and
propose to extend this by allowing a model to automatically (soft-)search for
parts of a source sentence that are relevant to predicting a target word,
without having to form these parts as a hard segment explicitly. With this new
approach, we achieve a translation performance comparable to the existing
state-of-the-art phrase-based system on the task of English-to-French
translation. Furthermore, qualitative analysis reveals that the
(soft-)alignments found by the model agree well with our intuition.
Dzmitry Bahdanau Kyunghyun Cho Yoshua Bengio