set. Note that the lengths of the sentences differ.
where
ij=exp (eij)PTx
k=1exp (eik)
eij=v>
atanh (Wasi1+Uahj);
andhjis thej-th annotation in the source sentence (see Eq. (7)). va2Rn0;Wa2Rn0nand
Ua2Rn02nare weight matrices. Note that the model becomes RNN Encoder–Decoder (Cho
et al. , 2014a), if we ﬁx cito !hTx.
With the decoder state si1, the context ciand the last generated word yi1, we deﬁne the probability
of a target word yias
p(yijsi;yi1;ci)/exp
y>
iWoti
;
where
ti=
max~ti;2j1;~ti;2j	>
j=1;:::;l
and~ti;kis thek-th element of a vector ~tiwhich is computed by
~ti=Uosi1+VoEyi1+Coci:
Neural machine translation is a recently proposed approach to machine
translation. Unlike the traditional statistical machine translation, the neural
machine translation aims at building a single neural network that can be
jointly tuned to maximize the translation performance. The models proposed
recently for neural machine translation often belong to a family of
encoder-decoders and consists of an encoder that encodes a source sentence into
a fixed-length vector from which a decoder generates a translation. In this
paper, we conjecture that the use of a fixed-length vector is a bottleneck in
improving the performance of this basic encoder-decoder architecture, and
propose to extend this by allowing a model to automatically (soft-)search for
parts of a source sentence that are relevant to predicting a target word,
without having to form these parts as a hard segment explicitly. With this new
approach, we achieve a translation performance comparable to the existing
state-of-the-art phrase-based system on the task of English-to-French
translation. Furthermore, qualitative analysis reveals that the
(soft-)alignments found by the model agree well with our intuition.
Dzmitry Bahdanau Kyunghyun Cho Yoshua Bengio