We can understand the approach of taking a weighted sum of all the annotations as computing an
expected annotation , where the expectation is over possible alignments. Let ijbe a probability that
the target word yiis aligned to, or translated from, a source word xj. Then, thei-th context vector
ciis the expected annotation over all the annotations with probabilities ij.
The probability ij, or its associated energy eij, reﬂects the importance of the annotation hjwith
respect to the previous hidden state si1in deciding the next state siand generating yi. Intuitively,
this implements a mechanism of attention in the decoder. The decoder decides parts of the source
sentence to pay attention to. By letting the decoder have an attention mechanism, we relieve the
encoder from the burden of having to encode all information in the source sentence into a ﬁxedlength vector. With this new approach the information can be spread throughout the sequence of
annotations, which can be selectively retrieved by the decoder accordingly.
3.2 E NCODER : BIDIRECTIONAL RNN FOR ANNOTATING SEQUENCES
The usual RNN, described in Eq. (1), reads an input sequence xin order starting from the ﬁrst
Neural machine translation is a recently proposed approach to machine
translation. Unlike the traditional statistical machine translation, the neural
machine translation aims at building a single neural network that can be
jointly tuned to maximize the translation performance. The models proposed
recently for neural machine translation often belong to a family of
encoder-decoders and consists of an encoder that encodes a source sentence into
a fixed-length vector from which a decoder generates a translation. In this
paper, we conjecture that the use of a fixed-length vector is a bottleneck in
improving the performance of this basic encoder-decoder architecture, and
propose to extend this by allowing a model to automatically (soft-)search for
parts of a source sentence that are relevant to predicting a target word,
without having to form these parts as a hard segment explicitly. With this new
approach, we achieve a translation performance comparable to the existing
state-of-the-art phrase-based system on the task of English-to-French
translation. Furthermore, qualitative analysis reveals that the
(soft-)alignments found by the model agree well with our intuition.
Dzmitry Bahdanau Kyunghyun Cho Yoshua Bengio