i2R(4dim (h`)+1)dim (h`)to denote state transition
parameters from layer ito layerj, and b2R4dim (h`)+1is a bias term. In the last layer L, the
2~z`
tcan also be implemented as a function of h`
t, e.g., ~z`
t=hard sigm (Uh`
t).
5
Published as a conference paper at ICLR 2017
1z`t1z`t1W``1z`1tz`t1h`+1t1h`t1h`+1th`th`1tU``+1U``W`+1`z`t
ˆxt+1g1tg2tg3th1th2th3thet
Figure 2: Left: The gating mechanism of the HM-RNN. Right: The output module when L= 3.
top-down connection is ignored, and we use h0
t=xt. Since the input should not be omitted, we
setz0
t= 1for allt. Also, we do not use the boundary detector for the last layer. The hard sigm is
deﬁned by hard sigm (x) = max
0;min
Learning both hierarchical and temporal representation has been among the
long-standing challenges of recurrent neural networks. Multiscale recurrent
neural networks have been considered as a promising approach to resolve this
issue, yet there has been a lack of empirical evidence showing that this type
of models can actually capture the temporal dependencies by discovering the
latent hierarchical structure of the sequence. In this paper, we propose a
novel multiscale approach, called the hierarchical multiscale recurrent neural
networks, which can capture the latent hierarchical structure in the sequence
by encoding the temporal dependencies with different timescales using a novel
update mechanism. We show some evidence that our proposed multiscale
architecture can discover underlying hierarchical structure in the sequences
without using explicit boundary information. We evaluate our proposed model on
character-level language modelling and handwriting sequence modelling.
Junyoung Chung Sungjin Ahn Yoshua Bengio