spatial data, learning both hierarchical and temporal representation has been among the long-standing
challenges of RNNs in spite of the fact that hierarchical multiscale structures naturally exist in many
temporal data (Schmidhuber, 1991; Mozer, 1993; El Hihi & Bengio, 1995; Lin et al., 1996; Koutník
et al., 2014).
A promising approach to model such hierarchical and temporal representation is the multiscale
RNNs (Schmidhuber, 1992; El Hihi & Bengio, 1995; Koutník et al., 2014). Based on the observation
that high-level abstraction changes slowly with temporal coherency while low-level abstraction
has quickly changing features sensitive to the precise local timing (El Hihi & Bengio, 1995), the
multiscale RNNs group hidden units into multiple modules of different timescales. In addition to
the fact that the architecture ﬁts naturally to the latent hierarchical structures in many temporal data,
the multiscale approach provides the following advantages that resolve some inherent problems
of standard RNNs: (a) computational efﬁciency obtained by updating the high-level layers less
frequently, (b) efﬁciently delivering long-term dependencies with fewer updates at the high-level
Learning both hierarchical and temporal representation has been among the
long-standing challenges of recurrent neural networks. Multiscale recurrent
neural networks have been considered as a promising approach to resolve this
issue, yet there has been a lack of empirical evidence showing that this type
of models can actually capture the temporal dependencies by discovering the
latent hierarchical structure of the sequence. In this paper, we propose a
novel multiscale approach, called the hierarchical multiscale recurrent neural
networks, which can capture the latent hierarchical structure in the sequence
by encoding the temporal dependencies with different timescales using a novel
update mechanism. We show some evidence that our proposed multiscale
architecture can discover underlying hierarchical structure in the sequences
without using explicit boundary information. We evaluate our proposed model on
character-level language modelling and handwriting sequence modelling.
Junyoung Chung Sungjin Ahn Yoshua Bengio