j; (7)
where
~hhti
j=
[Wx]j+
U
rhht1i
j
:(8)
In this formulation, when the reset gate is close
to 0, the hidden state is forced to ignore the previous hidden state and reset with the current input
1The LSTM unit, which has shown impressive results in
several applications such as speech recognition, has a memory cell and four gating units that adaptively control the information ﬂow inside the unit, compared to only two gating
units in the proposed hidden unit. For details on LSTM networks, see, e.g., (Graves, 2012).
z
rh h~xFigure 2: An illustration of the proposed hidden
activation function. The update gate zselects
whether the hidden state is to be updated with
a new hidden state ~h. The reset gate rdecides
whether the previous hidden state is ignored. See
Eqs. (5)–(8) for the detailed equations of r,z,h
and~h.
only. This effectively allows the hidden state to
drop any information that is found to be irrelevant
later in the future, thus, allowing a more compact
representation.
On the other hand, the update gate controls how
much information from the previous hidden state
In this paper, we propose a novel neural network model called RNN
Encoder-Decoder that consists of two recurrent neural networks (RNN). One RNN
encodes a sequence of symbols into a fixed-length vector representation, and
the other decodes the representation into another sequence of symbols. The
encoder and decoder of the proposed model are jointly trained to maximize the
conditional probability of a target sequence given a source sequence. The
performance of a statistical machine translation system is empirically found to
improve by using the conditional probabilities of phrase pairs computed by the
RNN Encoder-Decoder as an additional feature in the existing log-linear model.
Qualitatively, we show that the proposed model learns a semantically and
syntactically meaningful representation of linguistic phrases.
Kyunghyun Cho Bart van Merrienboer Caglar Gulcehre Dzmitry Bahdanau Fethi Bougares Holger Schwenk Yoshua Bengio