The usual RNN, described in Eq. (1), reads an input sequence xin order starting from the ﬁrst
symbolx1to the last one xTx. However, in the proposed scheme, we would like the annotation
of each word to summarize not only the preceding words, but also the following words. Hence,
we propose to use a bidirectional RNN (BiRNN, Schuster and Paliwal, 1997), which has been
successfully used recently in speech recognition (see, e.g., Graves et al. , 2013).
A BiRNN consists of forward and backward RNN’s. The forward RNN !freads the input sequence
as it is ordered (from x1toxTx) and calculates a sequence of forward hidden states ( !h1;; !hTx).
The backward RNN freads the sequence in the reverse order (from xTxtox1), resulting in a
sequence of backward hidden states ( h1;; hTx).
We obtain an annotation for each word xjby concatenating the forward hidden state !hjand the
backward one hj, i.e.,hj=h !h>
j; h>
ji>
Neural machine translation is a recently proposed approach to machine
translation. Unlike the traditional statistical machine translation, the neural
machine translation aims at building a single neural network that can be
jointly tuned to maximize the translation performance. The models proposed
recently for neural machine translation often belong to a family of
encoder-decoders and consists of an encoder that encodes a source sentence into
a fixed-length vector from which a decoder generates a translation. In this
paper, we conjecture that the use of a fixed-length vector is a bottleneck in
improving the performance of this basic encoder-decoder architecture, and
propose to extend this by allowing a model to automatically (soft-)search for
parts of a source sentence that are relevant to predicting a target word,
without having to form these parts as a hard segment explicitly. With this new
approach, we achieve a translation performance comparable to the existing
state-of-the-art phrase-based system on the task of English-to-French
translation. Furthermore, qualitative analysis reveals that the
(soft-)alignments found by the model agree well with our intuition.
Dzmitry Bahdanau Kyunghyun Cho Yoshua Bengio