and FLUSH operations. In order to implement these operations, we introduced a set of binary
variables and a novel update rule that is dependent on the states of these binary variables. Each binary
variable is learned to ﬁnd segments at its level, therefore, we call this binary variable, a boundary
detector. On the character-level language modelling, the HM-LSTM achieved state-of-the-art result
on the Text8 dataset and comparable results to the state-of-the-art results on the Penn Treebank
and Hutter Prize Wikipedia datasets. Also, the HM-LSTM outperformed the standard LSTM on
the handwriting sequence generation. Our results and analysis suggest that the proposed HM-RNN
can discover the latent hierarchical structure of the sequences and can learn efﬁcient hierarchical
multiscale representation that leads to better generalization performance.
3The plot function could be found at blog.otoro.net/2015/12/12/handwriting-generation-demo-in-tensorflow/ .
10
Published as a conference paper at ICLR 2017
ACKNOWLEDGMENTS
The authors would like to thank Alex Graves, Tom Schaul and Hado van Hasselt for their fruitful
comments and discussion. We acknowledge the support of the following agencies for research
Learning both hierarchical and temporal representation has been among the
long-standing challenges of recurrent neural networks. Multiscale recurrent
neural networks have been considered as a promising approach to resolve this
issue, yet there has been a lack of empirical evidence showing that this type
of models can actually capture the temporal dependencies by discovering the
latent hierarchical structure of the sequence. In this paper, we propose a
novel multiscale approach, called the hierarchical multiscale recurrent neural
networks, which can capture the latent hierarchical structure in the sequence
by encoding the temporal dependencies with different timescales using a novel
update mechanism. We show some evidence that our proposed multiscale
architecture can discover underlying hierarchical structure in the sequences
without using explicit boundary information. We evaluate our proposed model on
character-level language modelling and handwriting sequence modelling.
Junyoung Chung Sungjin Ahn Yoshua Bengio