data (Schmidhuber, 1991; 1992; Chung et al., 2015; 2016). However, considering the fact that
non-stationarity is prevalent in temporal data, and that many entities of abstraction such as words
and sentences are in variable length, we claim that it is important for an RNN to dynamically adapt
its timescales to the particulars of the input entities of various length. While this is trivial if the
hierarchical boundary structure is provided (Sordoni et al., 2015), it has been a challenge for an RNN
to discover the latent hierarchical structure in temporal data without explicit boundary information.
In this paper, we propose a novel multiscale RNN model, which can learn the hierarchical multiscale
structure from temporal data without explicit boundary information. This model, called a hierarchical
multiscale recurrent neural network (HM-RNN), does not assign ﬁxed update rates, but adaptively
determines proper update times corresponding to different abstraction levels of the layers. We ﬁnd
that this model tends to learn ﬁne timescales for low-level layers and coarse timescales for high-level
layers. To do this, we introduce a binary boundary detector at each layer. The boundary detector is
turned on only at the time steps where a segment of the corresponding abstraction level is completely
Learning both hierarchical and temporal representation has been among the
long-standing challenges of recurrent neural networks. Multiscale recurrent
neural networks have been considered as a promising approach to resolve this
issue, yet there has been a lack of empirical evidence showing that this type
of models can actually capture the temporal dependencies by discovering the
latent hierarchical structure of the sequence. In this paper, we propose a
novel multiscale approach, called the hierarchical multiscale recurrent neural
networks, which can capture the latent hierarchical structure in the sequence
by encoding the temporal dependencies with different timescales using a novel
update mechanism. We show some evidence that our proposed multiscale
architecture can discover underlying hierarchical structure in the sequences
without using explicit boundary information. We evaluate our proposed model on
character-level language modelling and handwriting sequence modelling.
Junyoung Chung Sungjin Ahn Yoshua Bengio