learning rate as in the Text8. In Table 1 (right), we show the HM-LSTM achieving the test BPC 1:32,
which is a tie with the state-of-the-art result among the neural models. Although the neural models,
show remarkable performances, their compression performance is still behind the best models such
as PAQ8hp12 (Mahoney, 2005) and decomp8 (Mahoney, 2009).
Visualizing Learned Hierarchical Multiscale Structure In Figure 3 and 4, we visualize the
boundaries detected by the boundary detectors of the HM-LSTM while reading a character sequence
of total length 270taken from the validation set of either the Penn Treebank or Hutter Prize Wikipedia
dataset. Due to the page width limit, the ﬁgure contains the sequence partitioned into three segments
of length 90. The white blocks indicate boundaries z`
t= 1 while the black blocks indicate the
non-boundaries z`
t= 0.
Interestingly in both ﬁgures, we can observe that the boundary detector of the ﬁrst layer, z1, tends
to be turned on when it sees a space or after it sees a space, which is a reasonable breakpoint to
separate between words. This is somewhat surprising because the model self-organizes this structure
8
Learning both hierarchical and temporal representation has been among the
long-standing challenges of recurrent neural networks. Multiscale recurrent
neural networks have been considered as a promising approach to resolve this
issue, yet there has been a lack of empirical evidence showing that this type
of models can actually capture the temporal dependencies by discovering the
latent hierarchical structure of the sequence. In this paper, we propose a
novel multiscale approach, called the hierarchical multiscale recurrent neural
networks, which can capture the latent hierarchical structure in the sequence
by encoding the temporal dependencies with different timescales using a novel
update mechanism. We show some evidence that our proposed multiscale
architecture can discover underlying hierarchical structure in the sequences
without using explicit boundary information. We evaluate our proposed model on
character-level language modelling and handwriting sequence modelling.
Junyoung Chung Sungjin Ahn Yoshua Bengio