output layer is modiﬁed to predict real-valued outputs. We use the mixture density network as the
output layer following Graves (2013), and use 400units for each HM-LSTM layer and for the output
embedding layer. In Table 3, we compare the log-likelihood averaged over the test sequences of the
IAM-OnDB dataset. We observe that the HM-LSTM outperforms the standard LSTM. The slope
annealing trick further improves the test log-likelihood of the HM-LSTM into 1167 in our setting. In
this experiment, we increased the slope awith the following schedule a= min (3 ;1 + 0:004Nepoch ).
In Figure 5, we let the HM-LSTM to read a randomly picked validation sequence and present the
visualization of handwriting examples by segments based on either the states of z2or the states of
pen-tip location3.
5 C ONCLUSION
In this paper, we proposed the HM-RNN that can capture the latent hierarchical structure of the
sequences. We introduced three types of operations to the RNN, which are the COPY , UPDATE
and FLUSH operations. In order to implement these operations, we introduced a set of binary
variables and a novel update rule that is dependent on the states of these binary variables. Each binary
Learning both hierarchical and temporal representation has been among the
long-standing challenges of recurrent neural networks. Multiscale recurrent
neural networks have been considered as a promising approach to resolve this
issue, yet there has been a lack of empirical evidence showing that this type
of models can actually capture the temporal dependencies by discovering the
latent hierarchical structure of the sequence. In this paper, we propose a
novel multiscale approach, called the hierarchical multiscale recurrent neural
networks, which can capture the latent hierarchical structure in the sequence
by encoding the temporal dependencies with different timescales using a novel
update mechanism. We show some evidence that our proposed multiscale
architecture can discover underlying hierarchical structure in the sequences
without using explicit boundary information. We evaluate our proposed model on
character-level language modelling and handwriting sequence modelling.
Junyoung Chung Sungjin Ahn Yoshua Bengio