Ebetween the forward and backward RNNs, unlike the weight matrices.
We concatenate the forward and backward states to to obtain the annotations (h1;h2;;hTx),
where
hi=" !hi hi#
(7)
A.2.2 D ECODER
The hidden state siof the decoder given the annotations from the encoder is computed by
si=(1zi)si1+zi~si;
where
~si= tanh (WEy i1+U[risi1] +Cci)
zi=(WzEyi1+Uzsi1+Czci)
ri=(WrEyi1+Ursi1+Crci)
Eis the word embedding matrix for the target language. W;W z;Wr2Rnm,U;Uz;Ur2Rnn,
andC;C z;Cr2Rn2nare weights. Again, mandnare the word embedding dimensionality
and the number of hidden units, respectively. The initial hidden state s0is computed by s0=
tanh
Ws h1
;whereWs2Rnn.
Neural machine translation is a recently proposed approach to machine
translation. Unlike the traditional statistical machine translation, the neural
machine translation aims at building a single neural network that can be
jointly tuned to maximize the translation performance. The models proposed
recently for neural machine translation often belong to a family of
encoder-decoders and consists of an encoder that encodes a source sentence into
a fixed-length vector from which a decoder generates a translation. In this
paper, we conjecture that the use of a fixed-length vector is a bottleneck in
improving the performance of this basic encoder-decoder architecture, and
propose to extend this by allowing a model to automatically (soft-)search for
parts of a source sentence that are relevant to predicting a target word,
without having to form these parts as a hard segment explicitly. With this new
approach, we achieve a translation performance comparable to the existing
state-of-the-art phrase-based system on the task of English-to-French
translation. Furthermore, qualitative analysis reveals that the
(soft-)alignments found by the model agree well with our intuition.
Dzmitry Bahdanau Kyunghyun Cho Yoshua Bengio