<tdenotes all previous
symbols at time t. We evaluate our model on three benchmark text corpora: (1) Penn Treebank, (2)
Text8 and (3) Hutter Prize Wikipedia. We use the bits-per-character (BPC), E[log2p(xt+1jxt)],
as the evaluation metric.
Model We use a model consisting of an input embedding layer, an RNN module and an output
module. The input embedding layer maps each input symbol into 128-dimensional continuous
vector without using any non-linearity. The RNN module is the HM-LSTM, described in Section 3,
with three layers. The output module is a feedforward neural network with two layers, an output
embedding layer and a softmax layer. Figure 2 (right) shows a diagram of the output module. At each
time step, the output embedding layer receives the hidden states of the three RNN layers as input. In
order to adaptively control the importance of each layer at each time step, we also introduce three
scalar gating units g`
t2Rto each of the layer outputs:
g`
t=sigm (w`[h1
t;;hL
t]); (11)
where w`2RPL
Learning both hierarchical and temporal representation has been among the
long-standing challenges of recurrent neural networks. Multiscale recurrent
neural networks have been considered as a promising approach to resolve this
issue, yet there has been a lack of empirical evidence showing that this type
of models can actually capture the temporal dependencies by discovering the
latent hierarchical structure of the sequence. In this paper, we propose a
novel multiscale approach, called the hierarchical multiscale recurrent neural
networks, which can capture the latent hierarchical structure in the sequence
by encoding the temporal dependencies with different timescales using a novel
update mechanism. We show some evidence that our proposed multiscale
architecture can discover underlying hierarchical structure in the sequences
without using explicit boundary information. We evaluate our proposed model on
character-level language modelling and handwriting sequence modelling.
Junyoung Chung Sungjin Ahn Yoshua Bengio