with the proposed RNN Encoder–Decoder in the
future.
4.4 Word and Phrase Representations
Since the proposed RNN Encoder–Decoder is not
speciﬁcally designed only for the task of machine
translation, here we brieﬂy look at the properties
of the trained model.
It has been known for some time that
continuous space language models using
neural networks are able to learn semantically meaningful embeddings (See, e.g.,
(Bengio et al., 2003; Mikolov et al., 2013)). Since
the proposed RNN Encoder–Decoder also projects
to and maps back from a sequence of words into
a continuous space vector, we expect to see a
similar property with the proposed model as well.
The left plot in Fig. 4 shows the 2–D embedding
of the words using the word embedding matrix
learned by the RNN Encoder–Decoder. The projection was done by the recently proposed BarnesHut-SNE (van der Maaten, 2013). We can clearly
see that semantically similar words are clusteredwith each other (see the zoomed-in plots in Fig. 4).
The proposed RNN Encoder–Decoder naturally
generates a continuous-space representation of a
phrase. The representation ( cin Fig. 1) in this
In this paper, we propose a novel neural network model called RNN
Encoder-Decoder that consists of two recurrent neural networks (RNN). One RNN
encodes a sequence of symbols into a fixed-length vector representation, and
the other decodes the representation into another sequence of symbols. The
encoder and decoder of the proposed model are jointly trained to maximize the
conditional probability of a target sequence given a source sequence. The
performance of a statistical machine translation system is empirically found to
improve by using the conditional probabilities of phrase pairs computed by the
RNN Encoder-Decoder as an additional feature in the existing log-linear model.
Qualitatively, we show that the proposed model learns a semantically and
syntactically meaningful representation of linguistic phrases.
Kyunghyun Cho Bart van Merrienboer Caglar Gulcehre Dzmitry Bahdanau Fethi Bougares Holger Schwenk Yoshua Bengio