Let us denote an source phrase by X= (x1;x2;:::;xN)and a target phrase by Y=
(y1;y2;:::;yM). Each phrase is a sequence of K-dimensional one-hot vectors, such that only one
element of the vector is 1and all the others are 0. The index of the active ( 1) element indicates the word
represented by the vector.
A.1 Encoder
Each word of the source phrase is embedded in a 500-dimensional vector space: e(xi)2R500.e(x)is
used in Sec. 4.4 to visualize the words.
The hidden state of an encoder consists of 1000 hidden units, and each one of them at time tis
computed by
hhti
j=zjhht1i
j + (1zj)~hhti
j;
where
~hhti
j= tanh
[We(xt)]j+
U
rhht1i
j
;
zj=
[Wze(xt)]j+
Uzhht1i
j
;
rj=
[Wre(xt)]j+
Urhht1i
j
:
In this paper, we propose a novel neural network model called RNN
Encoder-Decoder that consists of two recurrent neural networks (RNN). One RNN
encodes a sequence of symbols into a fixed-length vector representation, and
the other decodes the representation into another sequence of symbols. The
encoder and decoder of the proposed model are jointly trained to maximize the
conditional probability of a target sequence given a source sequence. The
performance of a statistical machine translation system is empirically found to
improve by using the conditional probabilities of phrase pairs computed by the
RNN Encoder-Decoder as an additional feature in the existing log-linear model.
Qualitatively, we show that the proposed model learns a semantically and
syntactically meaningful representation of linguistic phrases.
Kyunghyun Cho Bart van Merrienboer Caglar Gulcehre Dzmitry Bahdanau Fethi Bougares Holger Schwenk Yoshua Bengio