9
Published as a conference paper at ICLR 2017
IAM-OnDB
Model Average Log-Likelihood
Standard LSTM 1081
HM-LSTM 1137
HM-LSTM & Slope Annealing 1167
Table 3: Average log-likelihood per sequence on the IAM-OnDB test set.
the ground truth of pen-tip locationVisualization by segments usingVisualization by segments usingthe states ofz2
Figure 5: The visualization by segments based on either the given pen-tip location or states of the z2.
and only 9updates has made in the third layer. Note that, by design, the ﬁrst layer performs UPDATE
operation at every time step and then the number of UPDATE operations decreases as the layer level
increases. In this example, the total number of updates is 335for the HM-LSTM which is 60% of
reduction from the 810updates of the standard RNN architecture.
4.2 H ANDWRITING SEQUENCE GENERATION
We extend the evaluation of the HM-LSTM to a real-valued sequence modelling task using IAMOnDB (Liwicki & Bunke, 2005) dataset. The IAM-OnDB dataset consists of 12;179handwriting
Learning both hierarchical and temporal representation has been among the
long-standing challenges of recurrent neural networks. Multiscale recurrent
neural networks have been considered as a promising approach to resolve this
issue, yet there has been a lack of empirical evidence showing that this type
of models can actually capture the temporal dependencies by discovering the
latent hierarchical structure of the sequence. In this paper, we propose a
novel multiscale approach, called the hierarchical multiscale recurrent neural
networks, which can capture the latent hierarchical structure in the sequence
by encoding the temporal dependencies with different timescales using a novel
update mechanism. We show some evidence that our proposed multiscale
architecture can discover underlying hierarchical structure in the sequences
without using explicit boundary information. We evaluate our proposed model on
character-level language modelling and handwriting sequence modelling.
Junyoung Chung Sungjin Ahn Yoshua Bengio