j
;
rj=
[Wre(xt)]j+
Urhht1i
j
:
andare a logistic sigmoid function and an element-wise multiplication, respectively. To make the
equations uncluttered, we omit biases. The initial hidden state hh0i
jis ﬁxed to 0.
Once the hidden state at the Nstep (the end of the source phrase) is computed, the representation of
the source phrase cis
c= tanh
VhhNi
:
A.1.1 Decoder
The decoder starts by initializing the hidden state with
h0h0i= tanh
V0c
;
where we will use0to distinguish parameters of the decoder from those of the encoder.
The hidden state at time tof the decoder is computed by
h0hti
j=z0jh0ht1i
j + (1z0j)~h0hti
j;
where
~h0hti
j= tanh
W0e(yt1)
j+r0j
U0h0
ht1i+Cc
;
z0j=
In this paper, we propose a novel neural network model called RNN
Encoder-Decoder that consists of two recurrent neural networks (RNN). One RNN
encodes a sequence of symbols into a fixed-length vector representation, and
the other decodes the representation into another sequence of symbols. The
encoder and decoder of the proposed model are jointly trained to maximize the
conditional probability of a target sequence given a source sequence. The
performance of a statistical machine translation system is empirically found to
improve by using the conditional probabilities of phrase pairs computed by the
RNN Encoder-Decoder as an additional feature in the existing log-linear model.
Qualitatively, we show that the proposed model learns a semantically and
syntactically meaningful representation of linguistic phrases.
Kyunghyun Cho Bart van Merrienboer Caglar Gulcehre Dzmitry Bahdanau Fethi Bougares Holger Schwenk Yoshua Bengio