Although interpretation of the second layer boundaries is not as apparent as the ﬁrst layer boundaries,
it seems to segment at reasonable semantic / syntactic boundaries, e.g., “consumers may” - “want to
move their telephones a” - “little closer to the tv set <unk>”, and so on.
Another remarkable point is the fact that we do not pose any constraint on the number of boundaries
that the model can ﬁre up. The model, however, learns that it is more beneﬁcial to delay the
information ejection to some extent. This is somewhat counterintuitive because it might look more
beneﬁcial to feed the fresh update to the upper layers at every time step without any delay. We
conjecture the reason that the model works in this way is due to the FLUSH operation that poses an
implicit constraint on the frequency of boundary detection, because it contains both a reward (feeding
fresh information to upper layers) and a penalty (erasing accumulated information). The model ﬁnds
an optimal balance between the reward and the penalty.
To understand the update mechanism more intuitively, in Figure 4, we also depict the heatmap of the
`2-norm of the hidden states along with the states of the boundary detectors. As we expect, we can
Learning both hierarchical and temporal representation has been among the
long-standing challenges of recurrent neural networks. Multiscale recurrent
neural networks have been considered as a promising approach to resolve this
issue, yet there has been a lack of empirical evidence showing that this type
of models can actually capture the temporal dependencies by discovering the
latent hierarchical structure of the sequence. In this paper, we propose a
novel multiscale approach, called the hierarchical multiscale recurrent neural
networks, which can capture the latent hierarchical structure in the sequence
by encoding the temporal dependencies with different timescales using a novel
update mechanism. We show some evidence that our proposed multiscale
architecture can discover underlying hierarchical structure in the sequences
without using explicit boundary information. We evaluate our proposed model on
character-level language modelling and handwriting sequence modelling.
Junyoung Chung Sungjin Ahn Yoshua Bengio