score a pair of sequences (in terms of a conditional
probability) or generate a target sequence given a
source sequence. Along with the new architecture,
we proposed a novel hidden unit that includes a reset gate and an update gate that adaptively control
how much each hidden unit remembers or forgets
while reading/generating a sequence.
We evaluated the proposed model with the task
of statistical machine translation, where we used
the RNN Encoder–Decoder to score each phrase
pair in the phrase table. Qualitatively, we were
able to show that the new model is able to capture linguistic regularities in the phrase pairs well
and also that the RNN Encoder–Decoder is able to
propose well-formed target phrases.
The scores by the RNN Encoder–Decoder were
found to improve the overall translation performance in terms of BLEU scores. Also, we
found that the contribution by the RNN Encoder–
Decoder is rather orthogonal to the existing approach of using neural networks in the SMT system, so that we can improve further the performance by using, for instance, the RNN Encoder–
Decoder and the neural net language model together.
Our qualitative analysis of the trained model
shows that it indeed captures the linguistic regularities in multiple levels i.e. at the word level as
well as phrase level. This suggests that there may
In this paper, we propose a novel neural network model called RNN
Encoder-Decoder that consists of two recurrent neural networks (RNN). One RNN
encodes a sequence of symbols into a fixed-length vector representation, and
the other decodes the representation into another sequence of symbols. The
encoder and decoder of the proposed model are jointly trained to maximize the
conditional probability of a target sequence given a source sequence. The
performance of a statistical machine translation system is empirically found to
improve by using the conditional probabilities of phrase pairs computed by the
RNN Encoder-Decoder as an additional feature in the existing log-linear model.
Qualitatively, we show that the proposed model learns a semantically and
syntactically meaningful representation of linguistic phrases.
Kyunghyun Cho Bart van Merrienboer Caglar Gulcehre Dzmitry Bahdanau Fethi Bougares Holger Schwenk Yoshua Bengio