to obtain high-level representation of the sequences of reduced length by repeatedly merging or
pooling the lower-level representation of the sequences.
Hierarchical RNN architectures have also been used to discover the segmentation structure in
sequences (Fernández et al., 2007; Kong et al., 2015). It is however different to our model in the
sense that they optimize the objective with explicit labels on the hierarchical segments while our
model discovers the intrinsic structure only from the sequences without segment label information.
The COPY operation used in our model can be related to Zoneout (Krueger et al., 2016) which
is a recurrent generalization of stochastic depth (Huang et al., 2016). In Zoneout, an identity
transformation is randomly applied to each hidden unit at each time step according to a Bernoulli
distribution. This results in occasional copy operations of the previous hidden states. While the focus
of Zoneout is to propose a regularization technique similar to dropout (Srivastava et al., 2014) (where
the regularization strength is controlled by a hyperparameter), our model learns (a) to dynamically
determine when to copy from the context inputs and (b) to discover the hierarchical multiscale
structure and representation. Although the main goal of our proposed model is not regularization, we
found that our model also shows very good generalization performance.
Learning both hierarchical and temporal representation has been among the
long-standing challenges of recurrent neural networks. Multiscale recurrent
neural networks have been considered as a promising approach to resolve this
issue, yet there has been a lack of empirical evidence showing that this type
of models can actually capture the temporal dependencies by discovering the
latent hierarchical structure of the sequence. In this paper, we propose a
novel multiscale approach, called the hierarchical multiscale recurrent neural
networks, which can capture the latent hierarchical structure in the sequence
by encoding the temporal dependencies with different timescales using a novel
update mechanism. We show some evidence that our proposed multiscale
architecture can discover underlying hierarchical structure in the sequences
without using explicit boundary information. We evaluate our proposed model on
character-level language modelling and handwriting sequence modelling.
Junyoung Chung Sungjin Ahn Yoshua Bengio