Published as a conference paper at ICLR 2017
HIERARCHICAL MULTISCALE
RECURRENT NEURAL NETWORKS
Junyoung Chung, Sungjin Ahn & Yoshua Bengio
Département d’informatique et de recherche opérationnelle
Université de Montréal
{junyoung.chung,sungjin.ahn,yoshua.bengio}@umontreal.ca
ABSTRACT
Learning both hierarchical and temporal representation has been among the longstanding challenges of recurrent neural networks. Multiscale recurrent neural
networks have been considered as a promising approach to resolve this issue, yet
there has been a lack of empirical evidence showing that this type of models can
actually capture the temporal dependencies by discovering the latent hierarchical
structure of the sequence. In this paper, we propose a novel multiscale approach,
called the hierarchical multiscale recurrent neural network, that can capture the
latent hierarchical structure in the sequence by encoding the temporal dependencies
with different timescales using a novel update mechanism. We show some evidence
that the proposed model can discover underlying hierarchical structure in the
sequences without using explicit boundary information. We evaluate our proposed
model on character-level language modelling and handwriting sequence generation.
1 I NTRODUCTION
One of the key principles of learning in deep neural networks as well as in the human brain is to obtain
Learning both hierarchical and temporal representation has been among the
long-standing challenges of recurrent neural networks. Multiscale recurrent
neural networks have been considered as a promising approach to resolve this
issue, yet there has been a lack of empirical evidence showing that this type
of models can actually capture the temporal dependencies by discovering the
latent hierarchical structure of the sequence. In this paper, we propose a
novel multiscale approach, called the hierarchical multiscale recurrent neural
networks, which can capture the latent hierarchical structure in the sequence
by encoding the temporal dependencies with different timescales using a novel
update mechanism. We show some evidence that our proposed multiscale
architecture can discover underlying hierarchical structure in the sequences
without using explicit boundary information. We evaluate our proposed model on
character-level language modelling and handwriting sequence modelling.
Junyoung Chung Sungjin Ahn Yoshua Bengio