1 I NTRODUCTION
One of the key principles of learning in deep neural networks as well as in the human brain is to obtain
a hierarchical representation with increasing levels of abstraction (Bengio, 2009; LeCun et al., 2015;
Schmidhuber, 2015). A stack of representation layers, learned from the data in a way to optimize
the target task, make deep neural networks entertain advantages such as generalization to unseen
examples (Hoffman et al., 2013), sharing learned knowledge among multiple tasks, and discovering
disentangling factors of variation (Kingma & Welling, 2013). The remarkable recent successes of
the deep convolutional neural networks are particularly based on this ability to learn hierarchical
representation for spatial data (Krizhevsky et al., 2012). For modelling temporal data, the recent
resurgence of recurrent neural networks (RNN) has led to remarkable advances (Mikolov et al., 2010;
Graves, 2013; Cho et al., 2014; Sutskever et al., 2014; Vinyals et al., 2015). However, unlike the
spatial data, learning both hierarchical and temporal representation has been among the long-standing
challenges of RNNs in spite of the fact that hierarchical multiscale structures naturally exist in many
Learning both hierarchical and temporal representation has been among the
long-standing challenges of recurrent neural networks. Multiscale recurrent
neural networks have been considered as a promising approach to resolve this
issue, yet there has been a lack of empirical evidence showing that this type
of models can actually capture the temporal dependencies by discovering the
latent hierarchical structure of the sequence. In this paper, we propose a
novel multiscale approach, called the hierarchical multiscale recurrent neural
networks, which can capture the latent hierarchical structure in the sequence
by encoding the temporal dependencies with different timescales using a novel
update mechanism. We show some evidence that our proposed multiscale
architecture can discover underlying hierarchical structure in the sequences
without using explicit boundary information. We evaluate our proposed model on
character-level language modelling and handwriting sequence modelling.
Junyoung Chung Sungjin Ahn Yoshua Bengio