models are implemented by the authors to evaluate the performance using layer normalization (Ba
et al., 2016) with the additional output module. ( z) This method uses test error signals for predicting
the next characters, which makes it not comparable to other methods that do not.
sequence modelling, where the discrete symbols form a distinct hierarchical multiscale structure. The
performance on real-valued sequences is tested on the handwriting sequence generation in which a
relatively clear hierarchical multiscale structure exists compared to other data such as speech signals.
4.1 C HARACTER -LEVEL LANGUAGE MODELLING
A sequence modelling task aims at learning the probability distribution over sequences by minimizing
the negative log-likelihood of the training sequences:
min
1
NNX
n=1TnX
t=1logp(xn
tjxn
<t;); (10)
whereis the model parameter, Nis the number of training sequences, and Tnis the length of the
n-th sequence. A symbol at time tof sequence nis denoted by xn
t, andxn
<tdenotes all previous
symbols at time t. We evaluate our model on three benchmark text corpora: (1) Penn Treebank, (2)
Learning both hierarchical and temporal representation has been among the
long-standing challenges of recurrent neural networks. Multiscale recurrent
neural networks have been considered as a promising approach to resolve this
issue, yet there has been a lack of empirical evidence showing that this type
of models can actually capture the temporal dependencies by discovering the
latent hierarchical structure of the sequence. In this paper, we propose a
novel multiscale approach, called the hierarchical multiscale recurrent neural
networks, which can capture the latent hierarchical structure in the sequence
by encoding the temporal dependencies with different timescales using a novel
update mechanism. We show some evidence that our proposed multiscale
architecture can discover underlying hierarchical structure in the sequences
without using explicit boundary information. We evaluate our proposed model on
character-level language modelling and handwriting sequence modelling.
Junyoung Chung Sungjin Ahn Yoshua Bengio