BatchNorm LSTM (Cooijmans et al., 2016) 1.36
HM-LSTM 1.32
LayerNorm HM-LSTM 1.29
Table 2: BPC on the Text8 test set.
Penn Treebank We process the Penn Treebank dataset (Marcus et al., 1993) by following the
procedure introduced in Mikolov et al. (2012). Each update is done by using a mini-batch of 64
examples of length 100to prevent the memory overﬂow problem when unfolding the RNN in time
for backpropagation. The last hidden state of a sequence is used to initialize the hidden state of the
next sequence to approximate the full backpropagation. We train the model using Adam (Kingma &
Ba, 2014) with an initial learning rate of 0:002. We divide the learning rate by a factor of 50when
the validation negative log-likelihood stopped decreasing. The norm of the gradient is clipped with a
threshold of 1(Mikolov et al., 2010; Pascanu et al., 2012). We also apply layer normalization (Ba
et al., 2016) to our models. For all of the character-level language modelling experiments, we apply
the same procedure, but only change the number of hidden units, mini-batch size and the initial
learning rate.
Learning both hierarchical and temporal representation has been among the
long-standing challenges of recurrent neural networks. Multiscale recurrent
neural networks have been considered as a promising approach to resolve this
issue, yet there has been a lack of empirical evidence showing that this type
of models can actually capture the temporal dependencies by discovering the
latent hierarchical structure of the sequence. In this paper, we propose a
novel multiscale approach, called the hierarchical multiscale recurrent neural
networks, which can capture the latent hierarchical structure in the sequence
by encoding the temporal dependencies with different timescales using a novel
update mechanism. We show some evidence that our proposed multiscale
architecture can discover underlying hierarchical structure in the sequences
without using explicit boundary information. We evaluate our proposed model on
character-level language modelling and handwriting sequence modelling.
Junyoung Chung Sungjin Ahn Yoshua Bengio