frequently, (b) efﬁciently delivering long-term dependencies with fewer updates at the high-level
layers, which mitigates the vanishing gradient problem, (c) ﬂexible resource allocation (e.g., more
hidden units to the higher layers that focus on modelling long-term dependencies and less hidden
units to the lower layers which are in charge of learning short-term dependencies). In addition, the
learned latent hierarchical structures can provide useful information to other downstream tasks such
Yoshua Bengio is CIFAR Senior Fellow.
1arXiv:1609.01704v7  [cs.LG]  9 Mar 2017
Published as a conference paper at ICLR 2017
as module structures in computer program learning, sub-task structures in hierarchical reinforcement
learning, and story segments in video understanding.
There have been various approaches to implementing the multiscale RNNs. The most popular
approach is to set the timescales as hyperparameters (El Hihi & Bengio, 1995; Koutník et al., 2014;
Bahdanau et al., 2016) instead of treating them as dynamic variables that can be learned from the
data (Schmidhuber, 1991; 1992; Chung et al., 2015; 2016). However, considering the fact that
Learning both hierarchical and temporal representation has been among the
long-standing challenges of recurrent neural networks. Multiscale recurrent
neural networks have been considered as a promising approach to resolve this
issue, yet there has been a lack of empirical evidence showing that this type
of models can actually capture the temporal dependencies by discovering the
latent hierarchical structure of the sequence. In this paper, we propose a
novel multiscale approach, called the hierarchical multiscale recurrent neural
networks, which can capture the latent hierarchical structure in the sequence
by encoding the temporal dependencies with different timescales using a novel
update mechanism. We show some evidence that our proposed multiscale
architecture can discover underlying hierarchical structure in the sequences
without using explicit boundary information. We evaluate our proposed model on
character-level language modelling and handwriting sequence modelling.
Junyoung Chung Sungjin Ahn Yoshua Bengio