are the rows of a weight matrix W. By combining
these probabilities, we can compute the probability of the sequence xusing
p(x) =TY
t=1p(xtjxt1;:::;x 1): (3)
From this learned distribution, it is straightforward to sample a new sequence by iteratively sampling a symbol at each time step.
2.2 RNN Encoder–Decoder
In this paper, we propose a novel neural network
architecture that learns to encode a variable-length
sequence into a ﬁxed-length vector representation
and to decode a given ﬁxed-length vector representation back into a variable-length sequence.
From a probabilistic perspective, this new model
is a general method to learn the conditional distribution over a variable-length sequence conditioned on yet another variable-length sequence,
e.g.p(y1;:::;y T0jx1;:::;x T), where one
x1x2 xTyT' y2 y1
cDecoder
EncoderFigure 1: An illustration of the proposed RNN
Encoder–Decoder.
should note that the input and output sequence
lengthsTandT0may differ.
The encoder is an RNN that reads each symbol
of an input sequence xsequentially. As it reads
In this paper, we propose a novel neural network model called RNN
Encoder-Decoder that consists of two recurrent neural networks (RNN). One RNN
encodes a sequence of symbols into a fixed-length vector representation, and
the other decodes the representation into another sequence of symbols. The
encoder and decoder of the proposed model are jointly trained to maximize the
conditional probability of a target sequence given a source sequence. The
performance of a statistical machine translation system is empirically found to
improve by using the conditional probabilities of phrase pairs computed by the
RNN Encoder-Decoder as an additional feature in the existing log-linear model.
Qualitatively, we show that the proposed model learns a semantically and
syntactically meaningful representation of linguistic phrases.
Kyunghyun Cho Bart van Merrienboer Caglar Gulcehre Dzmitry Bahdanau Fethi Bougares Holger Schwenk Yoshua Bengio