decodes the representation into another sequence of symbols. The encoder and decoder of the proposed model are jointly
trained to maximize the conditional probability of a target sequence given a source
sequence. The performance of a statistical machine translation system is empirically found to improve by using the conditional probabilities of phrase pairs computed by the RNN Encoder–Decoder as an
additional feature in the existing log-linear
model. Qualitatively, we show that the
proposed model learns a semantically and
syntactically meaningful representation of
linguistic phrases.
1 Introduction
Deep neural networks have shown great success in
various applications such as objection recognition
(see, e.g., (Krizhevsky et al., 2012)) and speech
recognition (see, e.g., (Dahl et al., 2012)). Furthermore, many recent works showed that neural networks can be successfully used in a number of tasks in natural language processing (NLP).
These include, but are not limited to, language
modeling (Bengio et al., 2003), paraphrase detection (Socher et al., 2011) and word embedding extraction (Mikolov et al., 2013). In the ﬁeld of statistical machine translation (SMT), deep neural
networks have begun to show promising results.
(Schwenk, 2012) summarizes a successful usage
of feedforward neural networks in the framework
In this paper, we propose a novel neural network model called RNN
Encoder-Decoder that consists of two recurrent neural networks (RNN). One RNN
encodes a sequence of symbols into a fixed-length vector representation, and
the other decodes the representation into another sequence of symbols. The
encoder and decoder of the proposed model are jointly trained to maximize the
conditional probability of a target sequence given a source sequence. The
performance of a statistical machine translation system is empirically found to
improve by using the conditional probabilities of phrase pairs computed by the
RNN Encoder-Decoder as an additional feature in the existing log-linear model.
Qualitatively, we show that the proposed model learns a semantically and
syntactically meaningful representation of linguistic phrases.
Kyunghyun Cho Bart van Merrienboer Caglar Gulcehre Dzmitry Bahdanau Fethi Bougares Holger Schwenk Yoshua Bengio