train the model to learn the translation probability of an English phrase to a corresponding French
phrase. The model is then used as a part of a standard phrase-based SMT system by scoring each
phrase pair in the phrase table. The empirical evaluation reveals that this approach of scoring phrase
pairs with an RNN Encoder–Decoder improves
the translation performance.
We qualitatively analyze the trained RNN
Encoder–Decoder by comparing its phrase scores
with those given by the existing translation model.
The qualitative analysis shows that the RNN
Encoder–Decoder is better at capturing the linguistic regularities in the phrase table, indirectly
explaining the quantitative improvements in the
overall translation performance. The further analysis of the model reveals that the RNN Encoder–
Decoder learns a continuous space representation
of a phrase that preserves both the semantic and
syntactic structure of the phrase.arXiv:1406.1078v3  [cs.CL]  3 Sep 2014
2 RNN Encoder–Decoder
2.1 Preliminary: Recurrent Neural Networks
A recurrent neural network (RNN) is a neural network that consists of a hidden state hand an
optional output ywhich operates on a variablelength sequence x= (x1;:::;x T). At each time
stept, the hidden state hhtiof the RNN is updated
by
In this paper, we propose a novel neural network model called RNN
Encoder-Decoder that consists of two recurrent neural networks (RNN). One RNN
encodes a sequence of symbols into a fixed-length vector representation, and
the other decodes the representation into another sequence of symbols. The
encoder and decoder of the proposed model are jointly trained to maximize the
conditional probability of a target sequence given a source sequence. The
performance of a statistical machine translation system is empirically found to
improve by using the conditional probabilities of phrase pairs computed by the
RNN Encoder-Decoder as an additional feature in the existing log-linear model.
Qualitatively, we show that the proposed model learns a semantically and
syntactically meaningful representation of linguistic phrases.
Kyunghyun Cho Bart van Merrienboer Caglar Gulcehre Dzmitry Bahdanau Fethi Bougares Holger Schwenk Yoshua Bengio