separate between words. This is somewhat surprising because the model self-organizes this structure
8
Published as a conference paper at ICLR 2017
Figure 3: Hierarchical multiscale structure in the Wikipedia dataset captured by the boundary
detectors of the HM-LSTM.
Penn Treebank Line 1Penn Treebank Line 2Penn Treebank Line 3
Figure 4: The `2-norm of the hidden states shown together with the states of the boundary detectors
of the HM-LSTM.
without any explicit boundary information. In Figure 3, we observe that the z1tends to detect the
boundaries of the words but also ﬁres within the words, where the z2tends to ﬁre when it sees either
an end of a word or 2;3-grams. In Figure 4, we also see ﬂushing in the middle of a word, e.g.,
“tele-FLUSH-phone”. Note that “tele” is a preﬁx after which a various number of postﬁxes can follow.
From these, it seems that the model uses to some extent the concept of surprise to learn the boundary.
Although interpretation of the second layer boundaries is not as apparent as the ﬁrst layer boundaries,
Learning both hierarchical and temporal representation has been among the
long-standing challenges of recurrent neural networks. Multiscale recurrent
neural networks have been considered as a promising approach to resolve this
issue, yet there has been a lack of empirical evidence showing that this type
of models can actually capture the temporal dependencies by discovering the
latent hierarchical structure of the sequence. In this paper, we propose a
novel multiscale approach, called the hierarchical multiscale recurrent neural
networks, which can capture the latent hierarchical structure in the sequence
by encoding the temporal dependencies with different timescales using a novel
update mechanism. We show some evidence that our proposed multiscale
architecture can discover underlying hierarchical structure in the sequences
without using explicit boundary information. We evaluate our proposed model on
character-level language modelling and handwriting sequence modelling.
Junyoung Chung Sungjin Ahn Yoshua Bengio