for given activation functions fandg(the latter
must produce valid probabilities, e.g. with a softmax).
See Fig. 1 for a graphical depiction of the proposed model architecture.
The two components of the proposed RNN
Encoder–Decoder are jointly trained to maximize
the conditional log-likelihood
max
1
NNX
n=1logp(ynjxn); (4)
where is the set of the model parameters and
each (xn;yn)is an (input sequence, output sequence) pair from the training set. In our case,
as the output of the decoder, starting from the input, is differentiable, we can use a gradient-based
algorithm to estimate the model parameters.
Once the RNN Encoder–Decoder is trained, the
model can be used in two ways. One way is to use
the model to generate a target sequence given an
input sequence. On the other hand, the model can
be used to score a given pair of input and output
sequences, where the score is simply a probability
p(yjx)from Eqs. (3) and (4).
2.3 Hidden Unit that Adaptively Remembers
and Forgets
In addition to a novel model architecture, we also
propose a new type of hidden unit ( fin Eq. (1))
In this paper, we propose a novel neural network model called RNN
Encoder-Decoder that consists of two recurrent neural networks (RNN). One RNN
encodes a sequence of symbols into a fixed-length vector representation, and
the other decodes the representation into another sequence of symbols. The
encoder and decoder of the proposed model are jointly trained to maximize the
conditional probability of a target sequence given a source sequence. The
performance of a statistical machine translation system is empirically found to
improve by using the conditional probabilities of phrase pairs computed by the
RNN Encoder-Decoder as an additional feature in the existing log-linear model.
Qualitatively, we show that the proposed model learns a semantically and
syntactically meaningful representation of linguistic phrases.
Kyunghyun Cho Bart van Merrienboer Caglar Gulcehre Dzmitry Bahdanau Fethi Bougares Holger Schwenk Yoshua Bengio