network was trained to learn a mapping from a
bag-of-words representation of an input phrase to
an output phrase. This is closely related to both the
proposed RNN Encoder–Decoder and the model
proposed in (Schwenk, 2012), except that their input representation of a phrase is a bag-of-words.
A similar approach of using bag-of-words representations was proposed in (Gao et al., 2013) as
well. Earlier, a similar encoder–decoder model using two recursive neural networks was proposed
in (Socher et al., 2011), but their model was restricted to a monolingual setting, i.e. the model
reconstructs an input sentence. More recently, another encoder–decoder model using an RNN was
proposed in (Auli et al., 2013), where the decoder is conditioned on a representation of either
a source sentence or a source context.
One important difference between the proposed RNN Encoder–Decoder and the approaches
in (Zou et al., 2013) and (Chandar et al., 2014) is
that the order of the words in source and target phrases is taken into account. The RNN
Encoder–Decoder naturally distinguishes between
sequences that have the same words but in a different order, whereas the aforementioned approaches
effectively ignore order information.
The closest approach related to the proposed
In this paper, we propose a novel neural network model called RNN
Encoder-Decoder that consists of two recurrent neural networks (RNN). One RNN
encodes a sequence of symbols into a fixed-length vector representation, and
the other decodes the representation into another sequence of symbols. The
encoder and decoder of the proposed model are jointly trained to maximize the
conditional probability of a target sequence given a source sequence. The
performance of a statistical machine translation system is empirically found to
improve by using the conditional probabilities of phrase pairs computed by the
RNN Encoder-Decoder as an additional feature in the existing log-linear model.
Qualitatively, we show that the proposed model learns a semantically and
syntactically meaningful representation of linguistic phrases.
Kyunghyun Cho Bart van Merrienboer Caglar Gulcehre Dzmitry Bahdanau Fethi Bougares Holger Schwenk Yoshua Bengio