lengths: for example, different words have different lengths, so a ﬁxed hierarchy would not update its
upper-level units in synchrony with the natural boundaries in the data.
3.2 T HEPROPOSED MODEL
A key element of our model is the introduction of a parametrized boundary detector, which outputs
a binary value, in each layer of a stacked RNN, and learns when a segment should end in such
a way to optimize the overall target objective. Whenever the boundary detector is turned on at a
time step of layer `(i.e., when the boundary state is 1), the model considers this to be the end of a
1The acronym NARX stands for Non-linear Auto-Regressive model with eXogenous inputs.
4
Published as a conference paper at ICLR 2017
segment corresponding to the latent abstraction level of that layer (e.g., word or phrase) and feeds the
summarized representation of the detected segment into the upper layer ( `+ 1). Using the boundary
states, at each time step, each layer selects one of the following operations: UPDATE, COPY or
FLUSH. The selection is determined by (1) the boundary state of the current time step in the layer
belowz`1
tand (2) the boundary state of the previous time step in the same layer z`
t1.
Learning both hierarchical and temporal representation has been among the
long-standing challenges of recurrent neural networks. Multiscale recurrent
neural networks have been considered as a promising approach to resolve this
issue, yet there has been a lack of empirical evidence showing that this type
of models can actually capture the temporal dependencies by discovering the
latent hierarchical structure of the sequence. In this paper, we propose a
novel multiscale approach, called the hierarchical multiscale recurrent neural
networks, which can capture the latent hierarchical structure in the sequence
by encoding the temporal dependencies with different timescales using a novel
update mechanism. We show some evidence that our proposed multiscale
architecture can discover underlying hierarchical structure in the sequences
without using explicit boundary information. We evaluate our proposed model on
character-level language modelling and handwriting sequence modelling.
Junyoung Chung Sungjin Ahn Yoshua Bengio