commentary (5.5M), UN (421M) and two crawled corpora of 90M and 272.5M words respectively,
totaling 850M words. Following the procedure described in Cho et al. (2014a), we reduce the size of
the combined corpus to have 348M words using the data selection method by Axelrod et al. (2011).5
We do not use any monolingual data other than the mentioned parallel corpora, although it may be
possible to use a much larger monolingual corpus to pretrain an encoder. We concatenate news-test3http://www.statmt.org/wmt14/translation-task.html
4Implementations are available at https://github.com/lisa-groundhog/GroundHog .
5Available online at http://www-lium.univ-lemans.fr/ ˜schwenk/cslm_joint_paper/ .
4
Published as a conference paper at ICLR 2015
0 10 20 30 40 50 60
Sentence length051015202530BLEU score RNNsearch-50
RNNsearch-30
RNNenc-50
RNNenc-30
Figure 2: The BLEU scores
of the generated translations
on the test set with respect
to the lengths of the sentences. The results are on
Neural machine translation is a recently proposed approach to machine
translation. Unlike the traditional statistical machine translation, the neural
machine translation aims at building a single neural network that can be
jointly tuned to maximize the translation performance. The models proposed
recently for neural machine translation often belong to a family of
encoder-decoders and consists of an encoder that encodes a source sentence into
a fixed-length vector from which a decoder generates a translation. In this
paper, we conjecture that the use of a fixed-length vector is a bottleneck in
improving the performance of this basic encoder-decoder architecture, and
propose to extend this by allowing a model to automatically (soft-)search for
parts of a source sentence that are relevant to predicting a target word,
without having to form these parts as a hard segment explicitly. With this new
approach, we achieve a translation performance comparable to the existing
state-of-the-art phrase-based system on the task of English-to-French
translation. Furthermore, qualitative analysis reveals that the
(soft-)alignments found by the model agree well with our intuition.
Dzmitry Bahdanau Kyunghyun Cho Yoshua Bengio