=X
xj
t=2SLp
xj
tjx<t
p
xi
tjx<t
;
where x<tis a shorthand notation for xt1; : : : ; x 1.were not able to achieve better performance on the
test set, but only on the development set.
4.3 Qualitative Analysis
In order to understand where the performance improvement comes from, we analyze the phrase pair
scores computed by the RNN Encoder–Decoder
against the corresponding p(fje)from the translation model. Since the existing translation model
relies solely on the statistics of the phrase pairs in
the corpus, we expect its scores to be better estimated for the frequent phrases but badly estimated
for rare phrases. Also, as we mentioned earlier
in Sec. 3.1, we further expect the RNN Encoder–
Decoder which was trained without any frequency
information to score the phrase pairs based rather
on the linguistic regularities than on the statistics
of their occurrences in the corpus.
We focus on those pairs whose source phrase is
long (more than 3 words per source phrase) and
As a result, the probability of words not in the shortlist is
always overestimated. It is possible to address this issue by
backing off to an existing model that contain non-shortlisted
In this paper, we propose a novel neural network model called RNN
Encoder-Decoder that consists of two recurrent neural networks (RNN). One RNN
encodes a sequence of symbols into a fixed-length vector representation, and
the other decodes the representation into another sequence of symbols. The
encoder and decoder of the proposed model are jointly trained to maximize the
conditional probability of a target sequence given a source sequence. The
performance of a statistical machine translation system is empirically found to
improve by using the conditional probabilities of phrase pairs computed by the
RNN Encoder-Decoder as an additional feature in the existing log-linear model.
Qualitatively, we show that the proposed model learns a semantically and
syntactically meaningful representation of linguistic phrases.
Kyunghyun Cho Bart van Merrienboer Caglar Gulcehre Dzmitry Bahdanau Fethi Bougares Holger Schwenk Yoshua Bengio