pair in the original corpora. This measure was
taken in order (1) to reduce the computational expense of randomly selecting phrase pairs from a
large phrase table according to the normalized frequencies and (2) to ensure that the RNN Encoder–
Decoder does not simply learn to rank the phrase
pairs according to their numbers of occurrences.
One underlying reason for this choice was that the
existing translation probability in the phrase table already reﬂects the frequencies of the phrase
2Without loss of generality, from here on, we refer to
p(ejf)for each phrase pair as a translation model as wellpairs in the original corpus. With a ﬁxed capacity
of the RNN Encoder–Decoder, we try to ensure
that most of the capacity of the model is focused
toward learning linguistic regularities, i.e., distinguishing between plausible and implausible translations, or learning the “manifold” (region of probability concentration) of plausible translations.
Once the RNN Encoder–Decoder is trained, we
add a new score for each phrase pair to the existing phrase table. This allows the new scores to enter into the existing tuning algorithm with minimal
additional overhead in computation.
As Schwenk pointed out in (Schwenk, 2012),
it is possible to completely replace the existing
In this paper, we propose a novel neural network model called RNN
Encoder-Decoder that consists of two recurrent neural networks (RNN). One RNN
encodes a sequence of symbols into a fixed-length vector representation, and
the other decodes the representation into another sequence of symbols. The
encoder and decoder of the proposed model are jointly trained to maximize the
conditional probability of a target sequence given a source sequence. The
performance of a statistical machine translation system is empirically found to
improve by using the conditional probabilities of phrase pairs computed by the
RNN Encoder-Decoder as an additional feature in the existing log-linear model.
Qualitatively, we show that the proposed model learns a semantically and
syntactically meaningful representation of linguistic phrases.
Kyunghyun Cho Bart van Merrienboer Caglar Gulcehre Dzmitry Bahdanau Fethi Bougares Holger Schwenk Yoshua Bengio