Proceedings of the Empiricial Methods in Natural Language Processing (EMNLP 2014) , October 2014.
Junyoung Chung, Caglar Gulcehre, Kyunghyun Cho, and Yoshua Bengio. Gated feedback recurrent neural
networks. In Proceedings of the 32nd International Conference on Machine Learning (ICML) , 2015.
Junyoung Chung, Kyunghyun Cho, and Yoshua Bengio. A character-level decoder without explicit segmentation
for neural machine translation. Association for Computational Linguistics (ACL) , 2016.
Tim Cooijmans, Nicolas Ballas, César Laurent, and Aaron Courville. Recurrent batch normalization. arXiv
preprint arXiv:1603.09025 , 2016.
Matthieu Courbariaux, Itay Hubara, Daniel Soudry, Ran El-Yaniv, and Yoshua Bengio. Binarized neural
networks: Training deep neural networks with weights and activations constrained to+ 1 or-1. arXiv preprint
arXiv:1602.02830 , 2016.
Salah El Hihi and Yoshua Bengio. Hierarchical recurrent neural networks for long-term dependencies. In
Advances in Neural Information Processing Systems , pp. 493–499. Citeseer, 1995.
Learning both hierarchical and temporal representation has been among the
long-standing challenges of recurrent neural networks. Multiscale recurrent
neural networks have been considered as a promising approach to resolve this
issue, yet there has been a lack of empirical evidence showing that this type
of models can actually capture the temporal dependencies by discovering the
latent hierarchical structure of the sequence. In this paper, we propose a
novel multiscale approach, called the hierarchical multiscale recurrent neural
networks, which can capture the latent hierarchical structure in the sequence
by encoding the temporal dependencies with different timescales using a novel
update mechanism. We show some evidence that our proposed multiscale
architecture can discover underlying hierarchical structure in the sequences
without using explicit boundary information. We evaluate our proposed model on
character-level language modelling and handwriting sequence modelling.
Junyoung Chung Sungjin Ahn Yoshua Bengio