Table 1: BLEU scores computed on the development and test sets using different combinations of
approaches. WP denotes a word penalty , where
we penalizes the number of unknown words to
neural networks.
similarly. We used rank-100 matrices, equivalent
to learning an embedding of dimension 100 for
each word. The activation function used for ~hin
Eq. (8) is a hyperbolic tangent function. The computation from the hidden state in the decoder to
the output is implemented as a deep neural network (Pascanu et al., 2014) with a single intermediate layer having 500 maxout units each pooling
2 inputs (Goodfellow et al., 2013).
All the weight parameters in the RNN Encoder–
Decoder were initialized by sampling from an
isotropic zero-mean (white) Gaussian distribution
with its standard deviation ﬁxed to 0:01, except
for the recurrent weight parameters. For the recurrent weight matrices, we ﬁrst sampled from a
white Gaussian distribution and used its left singular vectors matrix, following (Saxe et al., 2014).
We used Adadelta and stochastic gradient
descent to train the RNN Encoder–Decoder
with hyperparameters = 106and=
0:95(Zeiler, 2012). At each update, we used 64
In this paper, we propose a novel neural network model called RNN
Encoder-Decoder that consists of two recurrent neural networks (RNN). One RNN
encodes a sequence of symbols into a fixed-length vector representation, and
the other decodes the representation into another sequence of symbols. The
encoder and decoder of the proposed model are jointly trained to maximize the
conditional probability of a target sequence given a source sequence. The
performance of a statistical machine translation system is empirically found to
improve by using the conditional probabilities of phrase pairs computed by the
RNN Encoder-Decoder as an additional feature in the existing log-linear model.
Qualitatively, we show that the proposed model learns a semantically and
syntactically meaningful representation of linguistic phrases.
Kyunghyun Cho Bart van Merrienboer Caglar Gulcehre Dzmitry Bahdanau Fethi Bougares Holger Schwenk Yoshua Bengio