sentence (x1;x2;:::;x T).In a new model architecture, we deﬁne each conditional probability
in Eq. (2) as:
p(yijy1;:::;y i1;x) =g(yi1;si;ci); (4)
wheresiis an RNN hidden state for time i, computed by
si=f(si1;yi1;ci):
It should be noted that unlike the existing encoder–decoder approach (see Eq. (2)), here the probability is conditioned on a distinct
context vector cifor each target word yi.
The context vector cidepends on a sequence of annotations
(h1;;hTx)to which an encoder maps the input sentence. Each
annotationhicontains information about the whole input sequence
with a strong focus on the parts surrounding the i-th word of the
input sequence. We explain in detail how the annotations are computed in the next section.
The context vector ciis, then, computed as a weighted sum of these
annotationshi:
ci=TxX
j=1ijhj: (5)
The weightijof each annotation hjis computed by
ij=exp (eij)PTx
Neural machine translation is a recently proposed approach to machine
translation. Unlike the traditional statistical machine translation, the neural
machine translation aims at building a single neural network that can be
jointly tuned to maximize the translation performance. The models proposed
recently for neural machine translation often belong to a family of
encoder-decoders and consists of an encoder that encodes a source sentence into
a fixed-length vector from which a decoder generates a translation. In this
paper, we conjecture that the use of a fixed-length vector is a bottleneck in
improving the performance of this basic encoder-decoder architecture, and
propose to extend this by allowing a model to automatically (soft-)search for
parts of a source sentence that are relevant to predicting a target word,
without having to form these parts as a hard segment explicitly. With this new
approach, we achieve a translation performance comparable to the existing
state-of-the-art phrase-based system on the task of English-to-French
translation. Furthermore, qualitative analysis reveals that the
(soft-)alignments found by the model agree well with our intuition.
Dzmitry Bahdanau Kyunghyun Cho Yoshua Bengio