layers. To do this, we introduce a binary boundary detector at each layer. The boundary detector is
turned on only at the time steps where a segment of the corresponding abstraction level is completely
processed. Otherwise, i.e., during the within segment processing, it stays turned off. Using the
hierarchical boundary states, we implement three operations, UPDATE, COPY and FLUSH, and
choose one of them at each time step. The UPDATE operation is similar to the usual update rule of
the long short-term memory (LSTM) (Hochreiter & Schmidhuber, 1997), except that it is executed
sparsely according to the detected boundaries. The COPY operation simply copies the cell and hidden
states of the previous time step. Unlike the leaky integration of the LSTM or the Gated Recurrent
Unit (GRU) (Cho et al., 2014), the COPY operation retains the whole states without any loss of
information. The FLUSH operation is executed when a boundary is detected, where it ﬁrst ejects the
summarized representation of the current segment to the upper layer and then reinitializes the states
to start processing the next segment. Learning to select a proper operation at each time step and to
detect the boundaries, the HM-RNN discovers the latent hierarchical structure of the sequences. We
Learning both hierarchical and temporal representation has been among the
long-standing challenges of recurrent neural networks. Multiscale recurrent
neural networks have been considered as a promising approach to resolve this
issue, yet there has been a lack of empirical evidence showing that this type
of models can actually capture the temporal dependencies by discovering the
latent hierarchical structure of the sequence. In this paper, we propose a
novel multiscale approach, called the hierarchical multiscale recurrent neural
networks, which can capture the latent hierarchical structure in the sequence
by encoding the temporal dependencies with different timescales using a novel
update mechanism. We show some evidence that our proposed multiscale
architecture can discover underlying hierarchical structure in the sequences
without using explicit boundary information. We evaluate our proposed model on
character-level language modelling and handwriting sequence modelling.
Junyoung Chung Sungjin Ahn Yoshua Bengio