RNNenc-30
Figure 2: The BLEU scores
of the generated translations
on the test set with respect
to the lengths of the sentences. The results are on
the full test set which includes sentences having unknown words to the models.
2012 and news-test-2013 to make a development (validation) set, and evaluate the models on the test
set (news-test-2014) from WMT ’14, which consists of 3003 sentences not present in the training
data.
After a usual tokenization6, we use a shortlist of 30,000 most frequent words in each language to
train our models. Any word not included in the shortlist is mapped to a special token ( [UNK ]). We
do not apply any other special preprocessing, such as lowercasing or stemming, to the data.
4.2 M ODELS
We train two types of models. The ﬁrst one is an RNN Encoder–Decoder (RNNencdec, Cho et al. ,
2014a), and the other is the proposed model, to which we refer as RNNsearch. We train each model
twice: ﬁrst with the sentences of length up to 30 words (RNNencdec-30, RNNsearch-30) and then
Neural machine translation is a recently proposed approach to machine
translation. Unlike the traditional statistical machine translation, the neural
machine translation aims at building a single neural network that can be
jointly tuned to maximize the translation performance. The models proposed
recently for neural machine translation often belong to a family of
encoder-decoders and consists of an encoder that encodes a source sentence into
a fixed-length vector from which a decoder generates a translation. In this
paper, we conjecture that the use of a fixed-length vector is a bottleneck in
improving the performance of this basic encoder-decoder architecture, and
propose to extend this by allowing a model to automatically (soft-)search for
parts of a source sentence that are relevant to predicting a target word,
without having to form these parts as a hard segment explicitly. With this new
approach, we achieve a translation performance comparable to the existing
state-of-the-art phrase-based system on the task of English-to-French
translation. Furthermore, qualitative analysis reveals that the
(soft-)alignments found by the model agree well with our intuition.
Dzmitry Bahdanau Kyunghyun Cho Yoshua Bengio