Other forms of Hierarchical RNN (HRNN) architectures have been proposed in the cases where
the explicit hierarchical boundary structure is provided. In Ling et al. (2015), after obtaining the
word boundary via tokenization, the HRNN architecture is used for neural machine translation by
modelling the characters and words using the ﬁrst and second RNN layers, respectively. A similar
HRNN architecture is also adopted in Sordoni et al. (2015) to model dialogue utterances. However,
in many cases, hierarchical boundary information is not explicitly observed or expensive to obtain.
Also, it is unclear how to deploy more layers than the number of boundary levels that is explicitly
observed in the data.
While the above models focus on online prediction problems, where a prediction needs to be made by
using only the past data, in some cases, predictions are made after observing the whole sequence. In
this setting, the input sequence can be regarded as 1-D spatial data, convolutional neural networks
with 1-D kernels are proposed in Kim (2014) and Kim et al. (2015) for language modelling and
sentence classiﬁcation. Also, in Chan et al. (2016) and Bahdanau et al. (2016), the authors proposed
to obtain high-level representation of the sequences of reduced length by repeatedly merging or
pooling the lower-level representation of the sequences.
Learning both hierarchical and temporal representation has been among the
long-standing challenges of recurrent neural networks. Multiscale recurrent
neural networks have been considered as a promising approach to resolve this
issue, yet there has been a lack of empirical evidence showing that this type
of models can actually capture the temporal dependencies by discovering the
latent hierarchical structure of the sequence. In this paper, we propose a
novel multiscale approach, called the hierarchical multiscale recurrent neural
networks, which can capture the latent hierarchical structure in the sequence
by encoding the temporal dependencies with different timescales using a novel
update mechanism. We show some evidence that our proposed multiscale
architecture can discover underlying hierarchical structure in the sequences
without using explicit boundary information. We evaluate our proposed model on
character-level language modelling and handwriting sequence modelling.
Junyoung Chung Sungjin Ahn Yoshua Bengio