networks have begun to show promising results.
(Schwenk, 2012) summarizes a successful usage
of feedforward neural networks in the framework
of phrase-based SMT system.Along this line of research on using neural networks for SMT, this paper focuses on a novel neural network architecture that can be used as a part
of the conventional phrase-based SMT system.
The proposed neural network architecture, which
we will refer to as an RNN Encoder–Decoder , consists of two recurrent neural networks (RNN) that
act as an encoder and a decoder pair. The encoder maps a variable-length source sequence to a
ﬁxed-length vector, and the decoder maps the vector representation back to a variable-length target
sequence. The two networks are trained jointly to
maximize the conditional probability of the target
sequence given a source sequence. Additionally,
we propose to use a rather sophisticated hidden
unit in order to improve both the memory capacity
and the ease of training.
The proposed RNN Encoder–Decoder with a
novel hidden unit is empirically evaluated on the
task of translating from English to French. We
train the model to learn the translation probability of an English phrase to a corresponding French
phrase. The model is then used as a part of a standard phrase-based SMT system by scoring each
In this paper, we propose a novel neural network model called RNN
Encoder-Decoder that consists of two recurrent neural networks (RNN). One RNN
encodes a sequence of symbols into a fixed-length vector representation, and
the other decodes the representation into another sequence of symbols. The
encoder and decoder of the proposed model are jointly trained to maximize the
conditional probability of a target sequence given a source sequence. The
performance of a statistical machine translation system is empirically found to
improve by using the conditional probabilities of phrase pairs computed by the
RNN Encoder-Decoder as an additional feature in the existing log-linear model.
Qualitatively, we show that the proposed model learns a semantically and
syntactically meaningful representation of linguistic phrases.
Kyunghyun Cho Bart van Merrienboer Caglar Gulcehre Dzmitry Bahdanau Fethi Bougares Holger Schwenk Yoshua Bengio