s0hti
2i1;s0hti
2io
and
s0hti=Ohh0hti+Oyyt1+Occ:
In short, the shti
iis a so-called maxout unit.
For the computational efﬁciency, instead of a single-matrix output weight G, we use a product of two
matrices such that
G=GlGr;
where Gl2RK500andGr2R5001000.
B Word and Phrase Representations
Here, we show enlarged plots of the word and phrase representations in Figs. 4–5.
Figure 6: 2–D embedding of the learned word representation. The top left one shows the full embedding space, while the other three ﬁgures show the zoomed-in
view of speciﬁc regions (color–coded).
Figure 7: 2–D embedding of the learned phrase representation. The top left one shows the full representation space (1000 randomly selected points), while the
other three ﬁgures show the zoomed-in view of speciﬁc regions (color–coded).
In this paper, we propose a novel neural network model called RNN
Encoder-Decoder that consists of two recurrent neural networks (RNN). One RNN
encodes a sequence of symbols into a fixed-length vector representation, and
the other decodes the representation into another sequence of symbols. The
encoder and decoder of the proposed model are jointly trained to maximize the
conditional probability of a target sequence given a source sequence. The
performance of a statistical machine translation system is empirically found to
improve by using the conditional probabilities of phrase pairs computed by the
RNN Encoder-Decoder as an additional feature in the existing log-linear model.
Qualitatively, we show that the proposed model learns a semantically and
syntactically meaningful representation of linguistic phrases.
Kyunghyun Cho Bart van Merrienboer Caglar Gulcehre Dzmitry Bahdanau Fethi Bougares Holger Schwenk Yoshua Bengio