additional overhead in computation.
As Schwenk pointed out in (Schwenk, 2012),
it is possible to completely replace the existing
phrase table with the proposed RNN Encoder–
Decoder. In that case, for a given source phrase,
the RNN Encoder–Decoder will need to generate
a list of (good) target phrases. This requires, however, an expensive sampling procedure to be performed repeatedly. In this paper, thus, we only
consider rescoring the phrase pairs in the phrase
table.
3.2 Related Approaches: Neural Networks in
Machine Translation
Before presenting the empirical results, we discuss
a number of recent works that have proposed to
use neural networks in the context of SMT.
Schwenk in (Schwenk, 2012) proposed a similar approach of scoring phrase pairs. Instead of the
RNN-based neural network, he used a feedforward
neural network that has ﬁxed-size inputs (7 words
in his case, with zero-padding for shorter phrases)
and ﬁxed-size outputs (7 words in the target language). When it is used speciﬁcally for scoring
phrases for the SMT system, the maximum phrase
length is often chosen to be small. However, as the
length of phrases increases or as we apply neural
In this paper, we propose a novel neural network model called RNN
Encoder-Decoder that consists of two recurrent neural networks (RNN). One RNN
encodes a sequence of symbols into a fixed-length vector representation, and
the other decodes the representation into another sequence of symbols. The
encoder and decoder of the proposed model are jointly trained to maximize the
conditional probability of a target sequence given a source sequence. The
performance of a statistical machine translation system is empirically found to
improve by using the conditional probabilities of phrase pairs computed by the
RNN Encoder-Decoder as an additional feature in the existing log-linear model.
Qualitatively, we show that the proposed model learns a semantically and
syntactically meaningful representation of linguistic phrases.
Kyunghyun Cho Bart van Merrienboer Caglar Gulcehre Dzmitry Bahdanau Fethi Bougares Holger Schwenk Yoshua Bengio