lengthsTandT0may differ.
The encoder is an RNN that reads each symbol
of an input sequence xsequentially. As it reads
each symbol, the hidden state of the RNN changes
according to Eq. (1). After reading the end of
the sequence (marked by an end-of-sequence symbol), the hidden state of the RNN is a summary c
of the whole input sequence.
The decoder of the proposed model is another
RNN which is trained to generate the output sequence by predicting the next symbol ytgiven the
hidden state hhti. However, unlike the RNN described in Sec. 2.1, both ytandhhtiare also conditioned onyt1and on the summary cof the input
sequence. Hence, the hidden state of the decoder
at timetis computed by,
hhti=f
hht1i;yt1;c
;
and similarly, the conditional distribution of the
next symbol is
P(ytjyt1;yt2;:::;y 1;c) =g
hhti;yt1;c
:
for given activation functions fandg(the latter
must produce valid probabilities, e.g. with a softmax).
See Fig. 1 for a graphical depiction of the proposed model architecture.
In this paper, we propose a novel neural network model called RNN
Encoder-Decoder that consists of two recurrent neural networks (RNN). One RNN
encodes a sequence of symbols into a fixed-length vector representation, and
the other decodes the representation into another sequence of symbols. The
encoder and decoder of the proposed model are jointly trained to maximize the
conditional probability of a target sequence given a source sequence. The
performance of a statistical machine translation system is empirically found to
improve by using the conditional probabilities of phrase pairs computed by the
RNN Encoder-Decoder as an additional feature in the existing log-linear model.
Qualitatively, we show that the proposed model learns a semantically and
syntactically meaningful representation of linguistic phrases.
Kyunghyun Cho Bart van Merrienboer Caglar Gulcehre Dzmitry Bahdanau Fethi Bougares Holger Schwenk Yoshua Bengio