2
Published as a conference paper at ICLR 2017
LSTMs (Hochreiter & Schmidhuber, 1997) employ the multiscale update concept, where the hidden
units have different forget and update rates and thus can operate with different timescales. However,
unlike our model, these timescales are not organized hierarchically. Although the LSTM has a selfloop for the gradients that helps to capture the long-term dependencies by mitigating the vanishing
gradient problem, in practice, it is still limited to a few hundred time steps due to the leaky integration
by which the contents to memorize for a long-term is gradually diluted at every time step. Also, the
model remains computationally expensive because it has to perform the update at every time step
for each unit. However, our model is less prone to these problems because it learns a hierarchical
structure such that, by design, high-level layers learn to perform less frequent updates than low-level
layers. We hypothesize that this property mitigates the vanishing gradient problem more efﬁciently
while also being computationally more efﬁcient.
A more recent model, the clockwork RNN (CW-RNN) (Koutník et al., 2014) extends the hierarchical
Learning both hierarchical and temporal representation has been among the
long-standing challenges of recurrent neural networks. Multiscale recurrent
neural networks have been considered as a promising approach to resolve this
issue, yet there has been a lack of empirical evidence showing that this type
of models can actually capture the temporal dependencies by discovering the
latent hierarchical structure of the sequence. In this paper, we propose a
novel multiscale approach, called the hierarchical multiscale recurrent neural
networks, which can capture the latent hierarchical structure in the sequence
by encoding the temporal dependencies with different timescales using a novel
update mechanism. We show some evidence that our proposed multiscale
architecture can discover underlying hierarchical structure in the sequences
without using explicit boundary information. We evaluate our proposed model on
character-level language modelling and handwriting sequence modelling.
Junyoung Chung Sungjin Ahn Yoshua Bengio