stept, the hidden state hhtiof the RNN is updated
by
hhti=f
hht1i;xt
; (1)
wherefis a non-linear activation function.fmay be as simple as an elementwise logistic sigmoid function and as complex as a long short-term memory (LSTM)
unit (Hochreiter and Schmidhuber, 1997).
An RNN can learn a probability distribution
over a sequence by being trained to predict the
next symbol in a sequence. In that case, the output
at each timestep tis the conditional distribution
p(xtjxt1;:::;x 1). For example, a multinomial
distribution ( 1-of-Kcoding) can be output using a
softmax activation function
p(xt;j= 1jxt1;:::;x 1) =exp
wjhhti
PK
j0=1exp
wj0hhti;
(2)
for all possible symbols j= 1;:::;K , where wj
are the rows of a weight matrix W. By combining
these probabilities, we can compute the probability of the sequence xusing
p(x) =TY
In this paper, we propose a novel neural network model called RNN
Encoder-Decoder that consists of two recurrent neural networks (RNN). One RNN
encodes a sequence of symbols into a fixed-length vector representation, and
the other decodes the representation into another sequence of symbols. The
encoder and decoder of the proposed model are jointly trained to maximize the
conditional probability of a target sequence given a source sequence. The
performance of a statistical machine translation system is empirically found to
improve by using the conditional probabilities of phrase pairs computed by the
RNN Encoder-Decoder as an additional feature in the existing log-linear model.
Qualitatively, we show that the proposed model learns a semantically and
syntactically meaningful representation of linguistic phrases.
Kyunghyun Cho Bart van Merrienboer Caglar Gulcehre Dzmitry Bahdanau Fethi Bougares Holger Schwenk Yoshua Bengio