The proposed RNN Encoder–Decoder naturally
generates a continuous-space representation of a
phrase. The representation ( cin Fig. 1) in this
case is a 1000-dimensional vector. Similarly to the
word representations, we visualize the representations of the phrases that consists of four or more
words using the Barnes-Hut-SNE in Fig. 5.
From the visualization, it is clear that the RNN
Encoder–Decoder captures both semantic and syntactic structures of the phrases. For instance, in
the bottom-left plot, most of the phrases are about
the duration of time, while those phrases that are
syntactically similar are clustered together. The
bottom-right plot shows the cluster of phrases that
are semantically similar (countries or regions). On
the other hand, the top-right plot shows the phrases
that are syntactically similar.
5 Conclusion
In this paper, we proposed a new neural network
architecture, called an RNN Encoder–Decoder
that is able to learn the mapping from a sequence
of an arbitrary length to another sequence, possibly from a different set, of an arbitrary length. The
proposed RNN Encoder–Decoder is able to either
score a pair of sequences (in terms of a conditional
probability) or generate a target sequence given a
source sequence. Along with the new architecture,
In this paper, we propose a novel neural network model called RNN
Encoder-Decoder that consists of two recurrent neural networks (RNN). One RNN
encodes a sequence of symbols into a fixed-length vector representation, and
the other decodes the representation into another sequence of symbols. The
encoder and decoder of the proposed model are jointly trained to maximize the
conditional probability of a target sequence given a source sequence. The
performance of a statistical machine translation system is empirically found to
improve by using the conditional probabilities of phrase pairs computed by the
RNN Encoder-Decoder as an additional feature in the existing log-linear model.
Qualitatively, we show that the proposed model learns a semantically and
syntactically meaningful representation of linguistic phrases.
Kyunghyun Cho Bart van Merrienboer Caglar Gulcehre Dzmitry Bahdanau Fethi Bougares Holger Schwenk Yoshua Bengio