of computation can be saved, (2) gradients are backpropagated through a much smaller number of
time steps, and (3) layer-wise capacity control becomes possible (e.g., use a smaller number of hidden
units in the ﬁrst layer which models short-term dependencies but whose updates are invoked much
more often).
Can an RNN discover such hierarchical multiscale structure without explicit hierarchical boundary
information? Considering the fact that the boundary information is difﬁcult to obtain (for example,
consider languages where words are not always cleanly separated by spaces or punctuation symbols,
and imperfect rules are used to separately perform segmentation) or usually not provided at all, this is
a legitimate problem. It gets worse when we consider higher-level concepts which we would like
the RNN to discover autonomously. In Section 2, we discussed the limitations of the existing RNN
models under this setting, which either have to update all units at every time step or use ﬁxed update
frequencies (El Hihi & Bengio, 1995; Koutník et al., 2014). Unfortunately, this kind of approach is
not well suited to the case where different segments in the hierarchical decomposition have different
lengths: for example, different words have different lengths, so a ﬁxed hierarchy would not update its
Learning both hierarchical and temporal representation has been among the
long-standing challenges of recurrent neural networks. Multiscale recurrent
neural networks have been considered as a promising approach to resolve this
issue, yet there has been a lack of empirical evidence showing that this type
of models can actually capture the temporal dependencies by discovering the
latent hierarchical structure of the sequence. In this paper, we propose a
novel multiscale approach, called the hierarchical multiscale recurrent neural
networks, which can capture the latent hierarchical structure in the sequence
by encoding the temporal dependencies with different timescales using a novel
update mechanism. We show some evidence that our proposed multiscale
architecture can discover underlying hierarchical structure in the sequences
without using explicit boundary information. We evaluate our proposed model on
character-level language modelling and handwriting sequence modelling.
Junyoung Chung Sungjin Ahn Yoshua Bengio