[Dahl et al.2012] George E. Dahl, Dong Yu, Li Deng,
and Alex Acero. 2012. Context-dependent pretrained deep neural networks for large vocabulary
speech recognition. IEEE Transactions on Audio,
Speech, and Language Processing , 20(1):33–42.
[Devlin et al.2014] Jacob Devlin, Rabih Zbib,
Zhongqiang Huang, Thomas Lamar, Richard
Schwartz, , and John Makhoul. 2014. Fast and
robust neural network joint models for statistical
machine translation. In Proceedings of the ACL
2014 Conference , ACL ’14, pages 1370–1380.
[Gao et al.2013] Jianfeng Gao, Xiaodong He, Wen tau
Yih, and Li Deng. 2013. Learning semantic representations for the phrase translation model. Technical report, Microsoft Research.
[Glorot et al.2011] X. Glorot, A. Bordes, and Y . Bengio. 2011. Deep sparse rectiﬁer neural networks. In
AISTATS’2011 .
[Goodfellow et al.2013] Ian J. Goodfellow, David
Warde-Farley, Mehdi Mirza, Aaron Courville, and
Yoshua Bengio. 2013. Maxout networks. In
ICML’2013 .
In this paper, we propose a novel neural network model called RNN
Encoder-Decoder that consists of two recurrent neural networks (RNN). One RNN
encodes a sequence of symbols into a fixed-length vector representation, and
the other decodes the representation into another sequence of symbols. The
encoder and decoder of the proposed model are jointly trained to maximize the
conditional probability of a target sequence given a source sequence. The
performance of a statistical machine translation system is empirically found to
improve by using the conditional probabilities of phrase pairs computed by the
RNN Encoder-Decoder as an additional feature in the existing log-linear model.
Qualitatively, we show that the proposed model learns a semantically and
syntactically meaningful representation of linguistic phrases.
Kyunghyun Cho Bart van Merrienboer Caglar Gulcehre Dzmitry Bahdanau Fethi Bougares Holger Schwenk Yoshua Bengio