deﬁned by hard sigm (x) = max
0;min
1;ax+1
2withabeing the slope variable.
Unlike the standard LSTM, the HM-LSTM has a top-down connection from (`+ 1) to`, which is
allowed to be activated only if a boundary is detected at the previous time step of the layer `(see
Eq. 6). This makes the layer `to be initialized with more long-term information after the boundary
is detected and execute the FLUSH operation. In addition, the input from the lower layer ( `1)
becomes effective only when a boundary is detected at the current time step in the layer ( `1) due
to the binary gate z`1
t. Figure 2 (left) shows the gating mechanism of the HM-LSTM at time step t.
Finally, the binary boundary state z`
tis obtained by:
z`
t=fbound(~z`
t): (8)
For the binarization function fbound :R!f0;1g, we can either use a deterministic step function:
z`
t=1if~z`
t>0:5
0otherwise;(9)
or sample from a Bernoulli distribution z`
tBernoulli (~z`
Learning both hierarchical and temporal representation has been among the
long-standing challenges of recurrent neural networks. Multiscale recurrent
neural networks have been considered as a promising approach to resolve this
issue, yet there has been a lack of empirical evidence showing that this type
of models can actually capture the temporal dependencies by discovering the
latent hierarchical structure of the sequence. In this paper, we propose a
novel multiscale approach, called the hierarchical multiscale recurrent neural
networks, which can capture the latent hierarchical structure in the sequence
by encoding the temporal dependencies with different timescales using a novel
update mechanism. We show some evidence that our proposed multiscale
architecture can discover underlying hierarchical structure in the sequences
without using explicit boundary information. We evaluate our proposed model on
character-level language modelling and handwriting sequence modelling.
Junyoung Chung Sungjin Ahn Yoshua Bengio