the baseline performance.
The best performance was achieved when we
used both CSLM and the phrase scores from the
RNN Encoder–Decoder. This suggests that the
contributions of the CSLM and the RNN Encoder–
Decoder are not too correlated and that one can
expect better results by improving each method independently. Furthermore, we tried penalizing the
number of words that are unknown to the neural
networks (i.e. words which are not in the shortlist). We do so by simply adding the number of
unknown words as an additional feature the loglinear model in Eq. (9).3However, in this case we
3To understand the effect of the penalty, consider the set
of all words in the 15,000 large shortlist, SL. All words xi=2
SL are replaced by a special token [UNK]before being scored
by the neural networks. Hence, the conditional probability of
anyxi
t=2SL is actually given by the model as
p(xt= [UNK]jx<t) =p(xt=2SLjx<t)
=X
xj
t=2SLp
xj
tjx<t
p
xi
tjx<t
;
In this paper, we propose a novel neural network model called RNN
Encoder-Decoder that consists of two recurrent neural networks (RNN). One RNN
encodes a sequence of symbols into a fixed-length vector representation, and
the other decodes the representation into another sequence of symbols. The
encoder and decoder of the proposed model are jointly trained to maximize the
conditional probability of a target sequence given a source sequence. The
performance of a statistical machine translation system is empirically found to
improve by using the conditional probabilities of phrase pairs computed by the
RNN Encoder-Decoder as an additional feature in the existing log-linear model.
Qualitatively, we show that the proposed model learns a semantically and
syntactically meaningful representation of linguistic phrases.
Kyunghyun Cho Bart van Merrienboer Caglar Gulcehre Dzmitry Bahdanau Fethi Bougares Holger Schwenk Yoshua Bengio