respectively denote the lengths of source and target sentences.
First, the forward states of the bidirectional recurrent neural network (BiRNN) are computed:
 !hi=(
(1 !zi) !hi1+ !zi !hi, ifi>0
0 , ifi= 0
where
 !hi= tanh !WExi+ !Uh !ri !hi1i
 !zi= !WzExi+ !Uz !hi1
 !ri= !WrExi+ !Ur !hi1
:
E2RmKxis the word embedding matrix. !W; !Wz; !Wr2Rnm, !U; !Uz; !Ur2Rnnare
weight matrices. mandnare the word embedding dimensionality and the number of hidden units,
respectively. ()is as usual a logistic sigmoid function.
The backward states ( h1;; hTx)are computed similarly. We share the word embedding matrix
Ebetween the forward and backward RNNs, unlike the weight matrices.
Neural machine translation is a recently proposed approach to machine
translation. Unlike the traditional statistical machine translation, the neural
machine translation aims at building a single neural network that can be
jointly tuned to maximize the translation performance. The models proposed
recently for neural machine translation often belong to a family of
encoder-decoders and consists of an encoder that encodes a source sentence into
a fixed-length vector from which a decoder generates a translation. In this
paper, we conjecture that the use of a fixed-length vector is a bottleneck in
improving the performance of this basic encoder-decoder architecture, and
propose to extend this by allowing a model to automatically (soft-)search for
parts of a source sentence that are relevant to predicting a target word,
without having to form these parts as a hard segment explicitly. With this new
approach, we achieve a translation performance comparable to the existing
state-of-the-art phrase-based system on the task of English-to-French
translation. Furthermore, qualitative analysis reveals that the
(soft-)alignments found by the model agree well with our intuition.
Dzmitry Bahdanau Kyunghyun Cho Yoshua Bengio