accordingly to maximize the BLEU score.
Since the neural net language model was proposed in (Bengio et al., 2003), neural networks
have been used widely in SMT systems. In
many cases, neural networks have been used to
rescore translation hypotheses ( n-best lists) (see,
e.g., (Schwenk et al., 2006)). Recently, however,
there has been interest in training neural networks
to score the translated sentence (or phrase pairs)
using a representation of the source sentence as
an additional input. See, e.g., (Schwenk, 2012),
(Son et al., 2012) and (Zou et al., 2013).
3.1 Scoring Phrase Pairs with RNN
Encoder–Decoder
Here we propose to train the RNN Encoder–
Decoder (see Sec. 2.2) on a table of phrase pairs
and use its scores as additional features in the loglinear model in Eq. (9) when tuning the SMT decoder.
When we train the RNN Encoder–Decoder, we
ignore the (normalized) frequencies of each phrase
pair in the original corpora. This measure was
taken in order (1) to reduce the computational expense of randomly selecting phrase pairs from a
In this paper, we propose a novel neural network model called RNN
Encoder-Decoder that consists of two recurrent neural networks (RNN). One RNN
encodes a sequence of symbols into a fixed-length vector representation, and
the other decodes the representation into another sequence of symbols. The
encoder and decoder of the proposed model are jointly trained to maximize the
conditional probability of a target sequence given a source sequence. The
performance of a statistical machine translation system is empirically found to
improve by using the conditional probabilities of phrase pairs computed by the
RNN Encoder-Decoder as an additional feature in the existing log-linear model.
Qualitatively, we show that the proposed model learns a semantically and
syntactically meaningful representation of linguistic phrases.
Kyunghyun Cho Bart van Merrienboer Caglar Gulcehre Dzmitry Bahdanau Fethi Bougares Holger Schwenk Yoshua Bengio