a sequence without using explicit boundary information.
We show that it is beneﬁcial to utilize the above structure through empirical evaluation.
We show that the straight-through estimator is an efﬁcient way of training a model containing
discrete variables.
We propose the slope annealing trick to improve the training procedure based on the
straight-through estimator.
2 R ELATED WORK
Two notable early attempts inspiring our model are Schmidhuber (1992) and El Hihi & Bengio (1995).
In these works, it is advocated to stack multiple layers of RNNs in a decreasing order of update
frequency for computational and learning efﬁciency. In Schmidhuber (1992), the author shows a
model that can self-organize a hierarchical multiscale structure. Particularly in El Hihi & Bengio
(1995), the advantages of incorporating a priori knowledge, “ temporal dependencies are structured
hierarchically ", into the RNN architecture is studied. The authors propose an RNN architecture that
updates each layer with a ﬁxed but different rate, called a hierarchical RNN.
2
Published as a conference paper at ICLR 2017
LSTMs (Hochreiter & Schmidhuber, 1997) employ the multiscale update concept, where the hidden
Learning both hierarchical and temporal representation has been among the
long-standing challenges of recurrent neural networks. Multiscale recurrent
neural networks have been considered as a promising approach to resolve this
issue, yet there has been a lack of empirical evidence showing that this type
of models can actually capture the temporal dependencies by discovering the
latent hierarchical structure of the sequence. In this paper, we propose a
novel multiscale approach, called the hierarchical multiscale recurrent neural
networks, which can capture the latent hierarchical structure in the sequence
by encoding the temporal dependencies with different timescales using a novel
update mechanism. We show some evidence that our proposed multiscale
architecture can discover underlying hierarchical structure in the sequences
without using explicit boundary information. We evaluate our proposed model on
character-level language modelling and handwriting sequence modelling.
Junyoung Chung Sungjin Ahn Yoshua Bengio