sentence e, which maximizes
p(fje)/p(ejf)p(f);
where the ﬁrst term at the right hand side is called
translation model and the latter language model
(see, e.g., (Koehn, 2005)). In practice, however,
most SMT systems model logp(fje)as a loglinear model with additional features and corresponding weights:
logp(fje) =NX
n=1wnfn(f;e) + logZ(e);(9)
wherefnandwnare then-th feature and weight,
respectively. Z(e)is a normalization constant that
does not depend on the weights. The weights are
often optimized to maximize the BLEU score on a
development set.
In the phrase-based SMT framework
introduced in (Koehn et al., 2003) and
(Marcu and Wong, 2002), the translation model
logp(ejf)is factorized into the translation
probabilities of matching phrases in the source
and target sentences.2These probabilities are
once again considered additional features in the
log-linear model (see Eq. (9)) and are weighted
accordingly to maximize the BLEU score.
Since the neural net language model was proposed in (Bengio et al., 2003), neural networks
In this paper, we propose a novel neural network model called RNN
Encoder-Decoder that consists of two recurrent neural networks (RNN). One RNN
encodes a sequence of symbols into a fixed-length vector representation, and
the other decodes the representation into another sequence of symbols. The
encoder and decoder of the proposed model are jointly trained to maximize the
conditional probability of a target sequence given a source sequence. The
performance of a statistical machine translation system is empirically found to
improve by using the conditional probabilities of phrase pairs computed by the
RNN Encoder-Decoder as an additional feature in the existing log-linear model.
Qualitatively, we show that the proposed model learns a semantically and
syntactically meaningful representation of linguistic phrases.
Kyunghyun Cho Bart van Merrienboer Caglar Gulcehre Dzmitry Bahdanau Fethi Bougares Holger Schwenk Yoshua Bengio