t1+i`
tg`
tifz`
t1= 0andz`1
t= 1(UPDATE)
c`
t1 ifz`
t1= 0andz`1
t= 0(COPY)
i`
tg`
t ifz`
t1= 1(FLUSH);(2)
and then the hidden state is obtained by:
h`
t=h`
t1 if COPY ,
o`
ttanh (c`
t)otherwise.(3)
Here, (f;i;o)are forget, input, output gates, and gis a cell proposal vector. Note that unlike the
LSTM, it is not necessary to compute these gates and cell proposal values at every time step. For
example, in the case of the COPY operation, we do not need to compute any of these values and thus
can save computations.
The COPY operation, which simply performs (c`
t;h`
t) (c`
t1;h`
t1), implements the observation
that an upper layer should keep its state unchanged until it receives the summarized input from
the lower layer. The UPDATE operation is performed to update the summary representation of the
layer`if the boundary z`1
Learning both hierarchical and temporal representation has been among the
long-standing challenges of recurrent neural networks. Multiscale recurrent
neural networks have been considered as a promising approach to resolve this
issue, yet there has been a lack of empirical evidence showing that this type
of models can actually capture the temporal dependencies by discovering the
latent hierarchical structure of the sequence. In this paper, we propose a
novel multiscale approach, called the hierarchical multiscale recurrent neural
networks, which can capture the latent hierarchical structure in the sequence
by encoding the temporal dependencies with different timescales using a novel
update mechanism. We show some evidence that our proposed multiscale
architecture can discover underlying hierarchical structure in the sequences
without using explicit boundary information. We evaluate our proposed model on
character-level language modelling and handwriting sequence modelling.
Junyoung Chung Sungjin Ahn Yoshua Bengio