A more recent model, the clockwork RNN (CW-RNN) (Koutník et al., 2014) extends the hierarchical
RNN (El Hihi & Bengio, 1995) and the NARX RNN (Lin et al., 1996)1. The CW-RNN tries to
solve the issue of using soft timescales in the LSTM, by explicitly assigning hard timescales. In the
CW-RNN, hidden units are partitioned into several modules, and different timescales are assigned to
the modules such that a module iupdates its hidden units at every 2(i1)-th time step. The CW-RNN
is computationally more efﬁcient than the standard RNN including the LSTM since hidden units
are updated only at the assigned clock rates. However, ﬁnding proper timescales in the CW-RNN
remains as a challenge whereas our model learns the intrinsic timescales from the data. In the biscale
RNNs (Chung et al., 2016), the authors proposed to model layer-wise timescales adaptively by having
additional gating units, however this approach still relies on the softgating mechanism like LSTMs.
Other forms of Hierarchical RNN (HRNN) architectures have been proposed in the cases where
Learning both hierarchical and temporal representation has been among the
long-standing challenges of recurrent neural networks. Multiscale recurrent
neural networks have been considered as a promising approach to resolve this
issue, yet there has been a lack of empirical evidence showing that this type
of models can actually capture the temporal dependencies by discovering the
latent hierarchical structure of the sequence. In this paper, we propose a
novel multiscale approach, called the hierarchical multiscale recurrent neural
networks, which can capture the latent hierarchical structure in the sequence
by encoding the temporal dependencies with different timescales using a novel
update mechanism. We show some evidence that our proposed multiscale
architecture can discover underlying hierarchical structure in the sequences
without using explicit boundary information. We evaluate our proposed model on
character-level language modelling and handwriting sequence modelling.
Junyoung Chung Sungjin Ahn Yoshua Bengio