the lower layer. The UPDATE operation is performed to update the summary representation of the
layer`if the boundary z`1
tis detected from the layer below but the boundary z`
t1was not found
at the previous time step. Hence, the UPDATE operation is executed sparsely unlike the standard
RNNs where it is executed at every time step, making it computationally inefﬁcient. If a boundary is
detected, the FLUSH operation is executed. The FLUSH operation consists of two sub-operations:
(a) EJECT to pass the current state to the upper layer and then (b) RESET to reinitialize the state
before starting to read a new segment. This operation implicitly forces the upper layer to absorb the
summary information of the lower layer segment, because otherwise it will be lost. Note that the
FLUSH operation is a hard reset in the sense that it completely erases all the previous states of the
same layer, which is different from the softreset or softforget operation in the GRU or LSTM.
Whenever needed (depending on the chosen operation), the gate values ( f`
t;i`
t;o`
t), the cell proposal
g`
t, and the pre-activation of the boundary detector ~z`
t2are then obtained by:
0
BBB@f`
t
i`
Learning both hierarchical and temporal representation has been among the
long-standing challenges of recurrent neural networks. Multiscale recurrent
neural networks have been considered as a promising approach to resolve this
issue, yet there has been a lack of empirical evidence showing that this type
of models can actually capture the temporal dependencies by discovering the
latent hierarchical structure of the sequence. In this paper, we propose a
novel multiscale approach, called the hierarchical multiscale recurrent neural
networks, which can capture the latent hierarchical structure in the sequence
by encoding the temporal dependencies with different timescales using a novel
update mechanism. We show some evidence that our proposed multiscale
architecture can discover underlying hierarchical structure in the sequences
without using explicit boundary information. We evaluate our proposed model on
character-level language modelling and handwriting sequence modelling.
Junyoung Chung Sungjin Ahn Yoshua Bengio