layer receives characters as inputs and generates word-level representations (C2W-RNN), and the
second layer takes the word-level representations as inputs and yields phrase-level representations
(W2P-RNN).
As shown, by means of the provided end-of-word labels, the C2W-RNN obtains word-level representation after processing the last character of each word and passes the word-level representation to the
W2P-RNN. Then, the W2P-RNN performs an update of the phrase-level representation. Note that the
hidden states of the W2P-RNN remains unchanged while all the characters of a word are processed by
the C2W-RNN. When the C2W-RNN starts to process the next word, its hidden states are reinitialized
using the latest hidden states of the W2P-RNN, which contain summarized representation of all the
words that have been processed by that time step, in that phrase.
From this simple example, we can see the advantages of having a hierarchical multiscale structure: (1)
as the W2P-RNN is updated at a much slower update rate than the C2W-RNN, a considerable amount
of computation can be saved, (2) gradients are backpropagated through a much smaller number of
Learning both hierarchical and temporal representation has been among the
long-standing challenges of recurrent neural networks. Multiscale recurrent
neural networks have been considered as a promising approach to resolve this
issue, yet there has been a lack of empirical evidence showing that this type
of models can actually capture the temporal dependencies by discovering the
latent hierarchical structure of the sequence. In this paper, we propose a
novel multiscale approach, called the hierarchical multiscale recurrent neural
networks, which can capture the latent hierarchical structure in the sequence
by encoding the temporal dependencies with different timescales using a novel
update mechanism. We show some evidence that our proposed multiscale
architecture can discover underlying hierarchical structure in the sequences
without using explicit boundary information. We evaluate our proposed model on
character-level language modelling and handwriting sequence modelling.
Junyoung Chung Sungjin Ahn Yoshua Bengio