2Although most of the previous works (see, e.g., Cho et al. , 2014a; Sutskever et al. , 2014; Kalchbrenner and
Blunsom, 2013) used to encode a variable-length input sentence into a ﬁxed-length vector, it is not necessary,
and even it may be beneﬁcial to have a variable-length vector, as we will show later.
2
Published as a conference paper at ICLR 2015
The decoder is often trained to predict the next word yt0given the context vector cand all the
previously predicted words fy1;;yt01g. In other words, the decoder deﬁnes a probability over
the translation yby decomposing the joint probability into the ordered conditionals:
p(y) =TY
t=1p(ytjfy1;;yt1g;c); (2)
where y=
y1;;yTy
. With an RNN, each conditional probability is modeled as
p(ytjfy1;;yt1g;c) =g(yt1;st;c); (3)
wheregis a nonlinear, potentially multi-layered, function that outputs the probability of yt, andstis
Neural machine translation is a recently proposed approach to machine
translation. Unlike the traditional statistical machine translation, the neural
machine translation aims at building a single neural network that can be
jointly tuned to maximize the translation performance. The models proposed
recently for neural machine translation often belong to a family of
encoder-decoders and consists of an encoder that encodes a source sentence into
a fixed-length vector from which a decoder generates a translation. In this
paper, we conjecture that the use of a fixed-length vector is a bottleneck in
improving the performance of this basic encoder-decoder architecture, and
propose to extend this by allowing a model to automatically (soft-)search for
parts of a source sentence that are relevant to predicting a target word,
without having to form these parts as a hard segment explicitly. With this new
approach, we achieve a translation performance comparable to the existing
state-of-the-art phrase-based system on the task of English-to-French
translation. Furthermore, qualitative analysis reveals that the
(soft-)alignments found by the model agree well with our intuition.
Dzmitry Bahdanau Kyunghyun Cho Yoshua Bengio