the same procedure, but only change the number of hidden units, mini-batch size and the initial
learning rate.
For the Penn Treebank dataset, we use 512units in each layer of the HM-LSTM and for the output
embedding layer. In Table 1 (left), we compare the test BPCs of four variants of our model to other
baseline models. Note that the HM-LSTM using the step function for the hard boundary decision
outperforms the others using either sampling orsoftboundary decision (i.e., hard sigmoid). The test
BPC is further improved with the slope annealing trick, which reduces the bias of the straight-through
estimator. We increased the slope awith the following schedule a= min (5 ;1 + 0:04Nepoch ), where
Nepoch is the maximum number of epochs. The HM-LSTM achieves test BPC score of 1:24. For the
remaining tasks, we ﬁxed the hard boundary decision using the step function without slope annealing
due to the difﬁculty of ﬁnding a good annealing schedule on large-scale datasets.
Text8 The Text8 dataset (Mahoney, 2009) consists of 100M characters extracted from the Wikipedia
Learning both hierarchical and temporal representation has been among the
long-standing challenges of recurrent neural networks. Multiscale recurrent
neural networks have been considered as a promising approach to resolve this
issue, yet there has been a lack of empirical evidence showing that this type
of models can actually capture the temporal dependencies by discovering the
latent hierarchical structure of the sequence. In this paper, we propose a
novel multiscale approach, called the hierarchical multiscale recurrent neural
networks, which can capture the latent hierarchical structure in the sequence
by encoding the temporal dependencies with different timescales using a novel
update mechanism. We show some evidence that our proposed multiscale
architecture can discover underlying hierarchical structure in the sequences
without using explicit boundary information. We evaluate our proposed model on
character-level language modelling and handwriting sequence modelling.
Junyoung Chung Sungjin Ahn Yoshua Bengio