backward one hj, i.e.,hj=h !h>
j; h>
ji>
. In this way, the annotation hjcontains the summaries
of both the preceding words and the following words. Due to the tendency of RNNs to better
represent recent inputs, the annotation hjwill be focused on the words around xj. This sequence
of annotations is used by the decoder and the alignment model later to compute the context vector
(Eqs. (5)–(6)).
See Fig. 1 for the graphical illustration of the proposed model.
4 E XPERIMENT SETTINGS
We evaluate the proposed approach on the task of English-to-French translation. We use the bilingual, parallel corpora provided by ACL WMT ’14.3As a comparison, we also report the performance of an RNN Encoder–Decoder which was proposed recently by Cho et al. (2014a). We use
the same training procedures and the same dataset for both models.4
4.1 D ATASET
WMT ’14 contains the following English-French parallel corpora: Europarl (61M words), news
commentary (5.5M), UN (421M) and two crawled corpora of 90M and 272.5M words respectively,
Neural machine translation is a recently proposed approach to machine
translation. Unlike the traditional statistical machine translation, the neural
machine translation aims at building a single neural network that can be
jointly tuned to maximize the translation performance. The models proposed
recently for neural machine translation often belong to a family of
encoder-decoders and consists of an encoder that encodes a source sentence into
a fixed-length vector from which a decoder generates a translation. In this
paper, we conjecture that the use of a fixed-length vector is a bottleneck in
improving the performance of this basic encoder-decoder architecture, and
propose to extend this by allowing a model to automatically (soft-)search for
parts of a source sentence that are relevant to predicting a target word,
without having to form these parts as a hard segment explicitly. With this new
approach, we achieve a translation performance comparable to the existing
state-of-the-art phrase-based system on the task of English-to-French
translation. Furthermore, qualitative analysis reveals that the
(soft-)alignments found by the model agree well with our intuition.
Dzmitry Bahdanau Kyunghyun Cho Yoshua Bengio