For training the neural networks, including the
proposed RNN Encoder–Decoder, we limited the
source and target vocabulary to the most frequent
15,000 words for both English and French. This
covers approximately 93% of the dataset. All the
out-of-vocabulary words were mapped to a special
token ( [UNK ]).
The baseline phrase-based SMT system was
built using Moses with default settings. This system achieves a BLEU score of 30.64 and 33.3 on
the development and test sets, respectively (see Table 1).
4.1.1 RNN Encoder–Decoder
The RNN Encoder–Decoder used in the experiment had 1000 hidden units with the proposed
gates at the encoder and at the decoder. The input matrix between each input symbol xhtiand the
hidden unit is approximated with two lower-rank
matrices, and the output matrix is approximated
ModelsBLEU
dev test
Baseline 30.64 33.30
RNN 31.20 33.87
CSLM + RNN 31.48 34.64
CSLM + RNN + WP 31.50 34.54
Table 1: BLEU scores computed on the development and test sets using different combinations of
approaches. WP denotes a word penalty , where
we penalizes the number of unknown words to
In this paper, we propose a novel neural network model called RNN
Encoder-Decoder that consists of two recurrent neural networks (RNN). One RNN
encodes a sequence of symbols into a fixed-length vector representation, and
the other decodes the representation into another sequence of symbols. The
encoder and decoder of the proposed model are jointly trained to maximize the
conditional probability of a target sequence given a source sequence. The
performance of a statistical machine translation system is empirically found to
improve by using the conditional probabilities of phrase pairs computed by the
RNN Encoder-Decoder as an additional feature in the existing log-linear model.
Qualitatively, we show that the proposed model learns a semantically and
syntactically meaningful representation of linguistic phrases.
Kyunghyun Cho Bart van Merrienboer Caglar Gulcehre Dzmitry Bahdanau Fethi Bougares Holger Schwenk Yoshua Bengio