structure and representation. Although the main goal of our proposed model is not regularization, we
found that our model also shows very good generalization performance.
3
Published as a conference paper at ICLR 2017
h20h22h25h15h14h13h12h11h10x1x2x3x4x5word1word2phrase1
h20h21h22h23h24h25h15h14h13h12h11h10x1x2x3x4x5word1word2phrase1
(a) (b)
Figure 1: (a) The HRNN architecture, which requires the knowledge of the hierarchical boundaries.
(b) The HM-RNN architecture that discovers the hierarchical multiscale structure in the data.
3 H IERARCHICAL MULTISCALE RECURRENT NEURAL NETWORKS
3.1 M OTIVATION
To begin with, we provide an example of how a stacked RNN can model temporal data in an ideal
setting, i.e., when the hierarchy of segments is provided (Sordoni et al., 2015; Ling et al., 2015). In
Figure 1 (a), we depict a hierarchical RNN (HRNN) for language modelling with two layers: the ﬁrst
layer receives characters as inputs and generates word-level representations (C2W-RNN), and the
Learning both hierarchical and temporal representation has been among the
long-standing challenges of recurrent neural networks. Multiscale recurrent
neural networks have been considered as a promising approach to resolve this
issue, yet there has been a lack of empirical evidence showing that this type
of models can actually capture the temporal dependencies by discovering the
latent hierarchical structure of the sequence. In this paper, we propose a
novel multiscale approach, called the hierarchical multiscale recurrent neural
networks, which can capture the latent hierarchical structure in the sequence
by encoding the temporal dependencies with different timescales using a novel
update mechanism. We show some evidence that our proposed multiscale
architecture can discover underlying hierarchical structure in the sequences
without using explicit boundary information. We evaluate our proposed model on
character-level language modelling and handwriting sequence modelling.
Junyoung Chung Sungjin Ahn Yoshua Bengio