sequences that have the same words but in a different order, whereas the aforementioned approaches
effectively ignore order information.
The closest approach related to the proposed
RNN Encoder–Decoder is the Recurrent Continuous Translation Model (Model 2) proposed in
(Kalchbrenner and Blunsom, 2013). In their paper, they proposed a similar model that consists
of an encoder and decoder. The difference with
our model is that they used a convolutional n-gram
model (CGM) for the encoder and the hybrid of
an inverse CGM and a recurrent neural network
for the decoder. They, however, evaluated their
model on rescoring the n-best list proposed by the
conventional SMT system and computing the perplexity of the gold standard translations.
4 Experiments
We evaluate our approach on the English/French
translation task of the WMT’14 workshop.4.1 Data and Baseline System
Large amounts of resources are available to build
an English/French SMT system in the framework
of the WMT’14 translation task. The bilingual
corpora include Europarl (61M words), news commentary (5.5M), UN (421M), and two crawled
corpora of 90M and 780M words respectively.
The last two corpora are quite noisy. To train
In this paper, we propose a novel neural network model called RNN
Encoder-Decoder that consists of two recurrent neural networks (RNN). One RNN
encodes a sequence of symbols into a fixed-length vector representation, and
the other decodes the representation into another sequence of symbols. The
encoder and decoder of the proposed model are jointly trained to maximize the
conditional probability of a target sequence given a source sequence. The
performance of a statistical machine translation system is empirically found to
improve by using the conditional probabilities of phrase pairs computed by the
RNN Encoder-Decoder as an additional feature in the existing log-linear model.
Qualitatively, we show that the proposed model learns a semantically and
syntactically meaningful representation of linguistic phrases.
Kyunghyun Cho Bart van Merrienboer Caglar Gulcehre Dzmitry Bahdanau Fethi Bougares Holger Schwenk Yoshua Bengio