later in the future, thus, allowing a more compact
representation.
On the other hand, the update gate controls how
much information from the previous hidden state
will carry over to the current hidden state. This
acts similarly to the memory cell in the LSTM
network and helps the RNN to remember longterm information. Furthermore, this may be considered an adaptive variant of a leaky-integration
unit (Bengio et al., 2013).
As each hidden unit has separate reset and update gates, each hidden unit will learn to capture
dependencies over different time scales. Those
units that learn to capture short-term dependencies
will tend to have reset gates that are frequently active, but those that capture longer-term dependencies will have update gates that are mostly active.
In our preliminary experiments, we found that
it is crucial to use this new unit with gating units.
We were not able to get meaningful result with an
oft-used tanh unit without any gating.
3 Statistical Machine Translation
In a commonly used statistical machine translation
system (SMT), the goal of the system (decoder,
speciﬁcally) is to ﬁnd a translation fgiven a source
sentence e, which maximizes
p(fje)/p(ejf)p(f);
where the ﬁrst term at the right hand side is called
In this paper, we propose a novel neural network model called RNN
Encoder-Decoder that consists of two recurrent neural networks (RNN). One RNN
encodes a sequence of symbols into a fixed-length vector representation, and
the other decodes the representation into another sequence of symbols. The
encoder and decoder of the proposed model are jointly trained to maximize the
conditional probability of a target sequence given a source sequence. The
performance of a statistical machine translation system is empirically found to
improve by using the conditional probabilities of phrase pairs computed by the
RNN Encoder-Decoder as an additional feature in the existing log-linear model.
Qualitatively, we show that the proposed model learns a semantically and
syntactically meaningful representation of linguistic phrases.
Kyunghyun Cho Bart van Merrienboer Caglar Gulcehre Dzmitry Bahdanau Fethi Bougares Holger Schwenk Yoshua Bengio