Text8 The Text8 dataset (Mahoney, 2009) consists of 100M characters extracted from the Wikipedia
corpus. Text8 contains only alphabets and spaces, and thus we have total 27 symbols. In order to
compare with other previous works, we follow the data splits used in Mikolov et al. (2012). We use
1024 units for each HM-LSTM layer and 2048 units for the output embedding layer. The mini-batch
size and the initial learning rate are set to 128and0:001, respectively. The results are shown in
Table 2. The HM-LSTM obtains the state-of-the-art test BPC 1:29.
Hutter Prize Wikipedia The Hutter Prize Wikipedia ( enwik8 ) dataset (Hutter, 2012) contains
205symbols including XML markups and special characters. We follow the data splits used in Graves
(2013) where the ﬁrst 90M characters are used to train the model, the next 5M characters for validation,
and the remainders for the test set. We use the same model size, mini-batch size and the initial
learning rate as in the Text8. In Table 1 (right), we show the HM-LSTM achieving the test BPC 1:32,
Learning both hierarchical and temporal representation has been among the
long-standing challenges of recurrent neural networks. Multiscale recurrent
neural networks have been considered as a promising approach to resolve this
issue, yet there has been a lack of empirical evidence showing that this type
of models can actually capture the temporal dependencies by discovering the
latent hierarchical structure of the sequence. In this paper, we propose a
novel multiscale approach, called the hierarchical multiscale recurrent neural
networks, which can capture the latent hierarchical structure in the sequence
by encoding the temporal dependencies with different timescales using a novel
update mechanism. We show some evidence that our proposed multiscale
architecture can discover underlying hierarchical structure in the sequences
without using explicit boundary information. We evaluate our proposed model on
character-level language modelling and handwriting sequence modelling.
Junyoung Chung Sungjin Ahn Yoshua Bengio