`2-norm of the hidden states along with the states of the boundary detectors. As we expect, we can
see that there is no change in the norm value within segments due to the COPY operation. Also, the
color ofkh1kchanges quickly (at every time step) because there is no COPY operation in the ﬁrst
layer. The color of kh2kchanges less frequently based on the states of z1
tandz2
t1. The color of
kh3kchanges even slowly, i.e., only when z2
t= 1.
A notable advantage of the proposed architecture is that the internal process of the RNN becomes
more interpretable. For example, we can substitute the states of z1
tandz2
t1into Eq. 2 and infer
which operation among the UPDATE, COPY and FLUSH was applied to the second layer at time step
t. We can also inspect the update frequencies of the layers simply by counting how many UPDATE
and FLUSH operations were made in each layer. For example in Figure 4, we see that the ﬁrst layer
updates at every time step (which is 270UPDATE operations), the second layer updates 56times,
9
Published as a conference paper at ICLR 2017
IAM-OnDB
Model Average Log-Likelihood
Standard LSTM 1081
Learning both hierarchical and temporal representation has been among the
long-standing challenges of recurrent neural networks. Multiscale recurrent
neural networks have been considered as a promising approach to resolve this
issue, yet there has been a lack of empirical evidence showing that this type
of models can actually capture the temporal dependencies by discovering the
latent hierarchical structure of the sequence. In this paper, we propose a
novel multiscale approach, called the hierarchical multiscale recurrent neural
networks, which can capture the latent hierarchical structure in the sequence
by encoding the temporal dependencies with different timescales using a novel
update mechanism. We show some evidence that our proposed multiscale
architecture can discover underlying hierarchical structure in the sequences
without using explicit boundary information. We evaluate our proposed model on
character-level language modelling and handwriting sequence modelling.
Junyoung Chung Sungjin Ahn Yoshua Bengio