examples, each of which is a sequence of (x;y)coordinate and a binary indicator pfor pen-tip
location, giving us (x1:Tn;y1:Tn;p1:Tn), wherenis an index of a sequence. At each time step,
the model receives (xt;yt;pt), and the goal is to predict (xt+1;yt+1;pt+1). The pen-up ( pt= 1)
indicates an end of a stroke, and the pen-down ( pt= 0) indicates that a stroke is in progress. There
is usually a large shift in the (x;y)coordinate to start a new stroke after the pen-up happens. We
remove all sequences whose length is shorter than 300. This leaves us 10;465sequences for training,
581for validation, 582for test. The average length of the sequences is 648. We normalize the range
of the (x;y)coordinates separately with the mean and standard deviation obtained from the training
set. We use the mini-batch size of 32, and the initial learning rate is set to 0:0003 .
We use the same model architecture as used in the character-level language model, except that the
output layer is modiﬁed to predict real-valued outputs. We use the mixture density network as the
Learning both hierarchical and temporal representation has been among the
long-standing challenges of recurrent neural networks. Multiscale recurrent
neural networks have been considered as a promising approach to resolve this
issue, yet there has been a lack of empirical evidence showing that this type
of models can actually capture the temporal dependencies by discovering the
latent hierarchical structure of the sequence. In this paper, we propose a
novel multiscale approach, called the hierarchical multiscale recurrent neural
networks, which can capture the latent hierarchical structure in the sequence
by encoding the temporal dependencies with different timescales using a novel
update mechanism. We show some evidence that our proposed multiscale
architecture can discover underlying hierarchical structure in the sequences
without using explicit boundary information. We evaluate our proposed model on
character-level language modelling and handwriting sequence modelling.
Junyoung Chung Sungjin Ahn Yoshua Bengio