~imikolov/rnnlm/char.pdf .
Andriy Mnih and Karol Gregor. Neural variational inference and learning in belief networks. In Proceedings of
the 31st International Conference on Machine Learning (ICML-14) , pp. 1791–1799, 2014.
Michael C Mozer. Induction of multiscale temporal structure. Advances in neural information processing
systems , pp. 275–275, 1993.
Vinod Nair and Geoffrey E Hinton. Rectiﬁed linear units improve restricted boltzmann machines. In Proceedings
of the 27th International Conference on Machine Learning (ICML-10) , pp. 807–814, 2010.
Marius Pachitariu and Maneesh Sahani. Regularization and nonlinearities for neural language models: when are
they needed? arXiv preprint arXiv:1301.5650 , 2013.
12
Published as a conference paper at ICLR 2017
Razvan Pascanu, Tomas Mikolov, and Yoshua Bengio. On the difﬁculty of training recurrent neural networks.
arXiv preprint arXiv:1211.5063 , 2012.
Kamil M Rocki. Recurrent memory array structures. arXiv preprint arXiv:1607.03085 , 2016a.
Learning both hierarchical and temporal representation has been among the
long-standing challenges of recurrent neural networks. Multiscale recurrent
neural networks have been considered as a promising approach to resolve this
issue, yet there has been a lack of empirical evidence showing that this type
of models can actually capture the temporal dependencies by discovering the
latent hierarchical structure of the sequence. In this paper, we propose a
novel multiscale approach, called the hierarchical multiscale recurrent neural
networks, which can capture the latent hierarchical structure in the sequence
by encoding the temporal dependencies with different timescales using a novel
update mechanism. We show some evidence that our proposed multiscale
architecture can discover underlying hierarchical structure in the sequences
without using explicit boundary information. We evaluate our proposed model on
character-level language modelling and handwriting sequence modelling.
Junyoung Chung Sungjin Ahn Yoshua Bengio