fed through two rectiﬁed layers (of size 1536 and
1024) (Glorot et al., 2011). The output layer was
a simple softmax layer (see Eq. (2)). All the
weight parameters were initialized uniformly between0:01and0:01, and the model was trained
until the validation perplexity did not improve for
10 epochs. After training, the language model
achieved a perplexity of 45.80. The validation set
was a random selection of 0.1% of the corpus. The
model was used to score partial translations during the decoding process, which generally leads to
higher gains in BLEU score than n-best list rescoring (Vaswani et al., 2013).
To address the computational complexity of
using a CSLM in the decoder a buffer was
used to aggregate n-grams during the stacksearch performed by the decoder. Only when
the buffer is full, or a stack is about to
be pruned, the n-grams are scored by the
CSLM. This allows us to perform fast matrixmatrix multiplication on GPU using Theano
(Bergstra et al., 2010; Bastien et al., 2012).
−60 −50 −40 −30 −20 −10 0−14−12−10−8−6−4−20
In this paper, we propose a novel neural network model called RNN
Encoder-Decoder that consists of two recurrent neural networks (RNN). One RNN
encodes a sequence of symbols into a fixed-length vector representation, and
the other decodes the representation into another sequence of symbols. The
encoder and decoder of the proposed model are jointly trained to maximize the
conditional probability of a target sequence given a source sequence. The
performance of a statistical machine translation system is empirically found to
improve by using the conditional probabilities of phrase pairs computed by the
RNN Encoder-Decoder as an additional feature in the existing log-linear model.
Qualitatively, we show that the proposed model learns a semantically and
syntactically meaningful representation of linguistic phrases.
Kyunghyun Cho Bart van Merrienboer Caglar Gulcehre Dzmitry Bahdanau Fethi Bougares Holger Schwenk Yoshua Bengio