always overestimated. It is possible to address this issue by
backing off to an existing model that contain non-shortlisted
words (see (Schwenk, 2007)) In this paper, however, we opt
for introducing a word penalty instead, which counteracts the
word probability overestimation.
Source Samples from RNN Encoder–Decoder
at the end of the [`a la ﬁn de la] (11)
for the ﬁrst time [pour la premi `ere fois] (24) [pour la premi `ere fois que] (2)
in the United States and [aux ´Etats-Unis et] (6) [dans les ´Etats-Unis et] (4)
, as well as [, ainsi que] [,] [ainsi que] [, ainsi qu’] [et UNK]
one of the most [l’ un des plus] (9) [l’ un des] (5) [l’ une des plus] (2)
(a) Long, frequent source phrases
Source Samples from RNN Encoder–Decoder
, Minister of Communications and Transport[ , ministre des communications et le transport] ( 13)
In this paper, we propose a novel neural network model called RNN
Encoder-Decoder that consists of two recurrent neural networks (RNN). One RNN
encodes a sequence of symbols into a fixed-length vector representation, and
the other decodes the representation into another sequence of symbols. The
encoder and decoder of the proposed model are jointly trained to maximize the
conditional probability of a target sequence given a source sequence. The
performance of a statistical machine translation system is empirically found to
improve by using the conditional probabilities of phrase pairs computed by the
RNN Encoder-Decoder as an additional feature in the existing log-linear model.
Qualitatively, we show that the proposed model learns a semantically and
syntactically meaningful representation of linguistic phrases.
Kyunghyun Cho Bart van Merrienboer Caglar Gulcehre Dzmitry Bahdanau Fethi Bougares Holger Schwenk Yoshua Bengio