proposed approach of training the RNN Encoder–
Decoder on a set of unique phrase pairs, discouraging the RNN Encoder–Decoder from learning
simply the frequencies of the phrase pairs from the
corpus, as explained earlier.
Furthermore, in Table 3, we show for each of
the source phrases in Table 2, the generated samples from the RNN Encoder–Decoder. For each
source phrase, we generated 50 samples and show
the top-ﬁve phrases accordingly to their scores.
We can see that the RNN Encoder–Decoder is
able to propose well-formed target phrases without looking at the actual phrase table. Importantly,
the generated phrases do not overlap completely
with the target phrases from the phrase table. This
encourages us to further investigate the possibility
of replacing the whole or a part of the phrase table
Figure 5: 2–D embedding of the learned phrase representation. The top left one shows the full representation space (5000 randomly selected points), while the other three ﬁgures show the zoomed-in view of
speciﬁc regions (color–coded).
with the proposed RNN Encoder–Decoder in the
future.
4.4 Word and Phrase Representations
Since the proposed RNN Encoder–Decoder is not
In this paper, we propose a novel neural network model called RNN
Encoder-Decoder that consists of two recurrent neural networks (RNN). One RNN
encodes a sequence of symbols into a fixed-length vector representation, and
the other decodes the representation into another sequence of symbols. The
encoder and decoder of the proposed model are jointly trained to maximize the
conditional probability of a target sequence given a source sequence. The
performance of a statistical machine translation system is empirically found to
improve by using the conditional probabilities of phrase pairs computed by the
RNN Encoder-Decoder as an additional feature in the existing log-linear model.
Qualitatively, we show that the proposed model learns a semantically and
syntactically meaningful representation of linguistic phrases.
Kyunghyun Cho Bart van Merrienboer Caglar Gulcehre Dzmitry Bahdanau Fethi Bougares Holger Schwenk Yoshua Bengio