and Forgets
In addition to a novel model architecture, we also
propose a new type of hidden unit ( fin Eq. (1))
that has been motivated by the LSTM unit but is
much simpler to compute and implement.1Fig. 2
shows the graphical depiction of the proposed hidden unit.
Let us describe how the activation of the j-th
hidden unit is computed. First, the reset gaterjis
computed by
rj=
[Wrx]j+
Urhht1i
j
; (5)
whereis the logistic sigmoid function, and [:]j
denotes the j-th element of a vector. xandht1
are the input and the previous hidden state, respectively. WrandUrare weight matrices which are
learned.
Similarly, the update gatezjis computed by
zj=
[Wzx]j+
Uzhht1i
j
: (6)
The actual activation of the proposed unit hjis
then computed by
hhti
j=zjhht1i
j + (1zj)~hhti
j; (7)
where
~hhti
j=
[Wx]j+
U
In this paper, we propose a novel neural network model called RNN
Encoder-Decoder that consists of two recurrent neural networks (RNN). One RNN
encodes a sequence of symbols into a fixed-length vector representation, and
the other decodes the representation into another sequence of symbols. The
encoder and decoder of the proposed model are jointly trained to maximize the
conditional probability of a target sequence given a source sequence. The
performance of a statistical machine translation system is empirically found to
improve by using the conditional probabilities of phrase pairs computed by the
RNN Encoder-Decoder as an additional feature in the existing log-linear model.
Qualitatively, we show that the proposed model learns a semantically and
syntactically meaningful representation of linguistic phrases.
Kyunghyun Cho Bart van Merrienboer Caglar Gulcehre Dzmitry Bahdanau Fethi Bougares Holger Schwenk Yoshua Bengio