to start processing the next segment. Learning to select a proper operation at each time step and to
detect the boundaries, the HM-RNN discovers the latent hierarchical structure of the sequences. We
ﬁnd that the straight-through estimator (Hinton, 2012; Bengio et al., 2013; Courbariaux et al., 2016)
is efﬁcient for training this model containing discrete variables.
We evaluate our model on two tasks: character-level language modelling and handwriting sequence
generation. For the character-level language modelling, the HM-RNN achieves the state-of-the-art
results on the Text8 dataset, and comparable results to the state-of-the-art on the Penn Treebank
and Hutter Prize Wikipedia datasets. The HM-RNN also outperforms the standard RNN on the
handwriting sequence generation using the IAM-OnDB dataset. In addition, we demonstrate that the
hierarchical structure found by the HM-RNN is indeed very similar to the intrinsic structure observed
in the data. The contributions of this paper are:
We propose for the ﬁrst time an RNN model that can learn a latent hierarchical structure of
a sequence without using explicit boundary information.
We show that it is beneﬁcial to utilize the above structure through empirical evaluation.
Learning both hierarchical and temporal representation has been among the
long-standing challenges of recurrent neural networks. Multiscale recurrent
neural networks have been considered as a promising approach to resolve this
issue, yet there has been a lack of empirical evidence showing that this type
of models can actually capture the temporal dependencies by discovering the
latent hierarchical structure of the sequence. In this paper, we propose a
novel multiscale approach, called the hierarchical multiscale recurrent neural
networks, which can capture the latent hierarchical structure in the sequence
by encoding the temporal dependencies with different timescales using a novel
update mechanism. We show some evidence that our proposed multiscale
architecture can discover underlying hierarchical structure in the sequences
without using explicit boundary information. We evaluate our proposed model on
character-level language modelling and handwriting sequence modelling.
Junyoung Chung Sungjin Ahn Yoshua Bengio