The authors would like to thank Alex Graves, Tom Schaul and Hado van Hasselt for their fruitful
comments and discussion. We acknowledge the support of the following agencies for research
funding and computing support: Ubisoft, Samsung, IBM, Facebook, Google, Microsoft, NSERC,
Calcul Québec, Compute Canada, the Canada Research Chairs and CIFAR. The authors thank the
developers of Theano (Team et al., 2016). JC would like to thank Arnaud Bergenon and Frédéric
Bastien for their technical support. JC would also like to thank Guillaume Alain, Kyle Kastner and
David Ha for providing us useful pieces of code.
REFERENCES
Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization. arXiv preprint arXiv:1607.06450 ,
2016.
Dzmitry Bahdanau, Jan Chorowski, Dmitriy Serdyuk, Yoshua Bengio, et al. End-to-end attention-based large
vocabulary speech recognition. In 2016 IEEE International Conference on Acoustics, Speech and Signal
Processing (ICASSP) , pp. 4945–4949. IEEE, 2016.
Learning both hierarchical and temporal representation has been among the
long-standing challenges of recurrent neural networks. Multiscale recurrent
neural networks have been considered as a promising approach to resolve this
issue, yet there has been a lack of empirical evidence showing that this type
of models can actually capture the temporal dependencies by discovering the
latent hierarchical structure of the sequence. In this paper, we propose a
novel multiscale approach, called the hierarchical multiscale recurrent neural
networks, which can capture the latent hierarchical structure in the sequence
by encoding the temporal dependencies with different timescales using a novel
update mechanism. We show some evidence that our proposed multiscale
architecture can discover underlying hierarchical structure in the sequences
without using explicit boundary information. We evaluate our proposed model on
character-level language modelling and handwriting sequence modelling.
Junyoung Chung Sungjin Ahn Yoshua Bengio