straight-through estimator, however, is much simpler and often works more efﬁciently in practice
than other unbiased but high-variance estimators such as the REINFORCE. The straight-through
estimator has also been used in Courbariaux et al. (2016) and Vezhnevets et al. (2016).
The Slope Annealing Trick . In our experiment, we use the slope annealing trick to reduce the bias
of the straight-through estimator. The idea is to reduce the discrepancy between the two functions
used during the forward pass and the backward pass. That is, by gradually increasing the slope aof
the hard sigmoid function, we make the hard sigmoid be close to the step function. Note that starting
with a high slope value from the beginning can make the training difﬁcult while it is more applicable
later when the model parameters become more stable. In our experiments, starting from slope a= 1,
we slowly increase the slope until it reaches a threshold with an appropriate scheduling.
4 E XPERIMENTS
We evaluate the proposed model on two tasks, character-level language modelling and handwriting
sequence generation. Character-level language modelling is a representative example of discrete
6
Published as a conference paper at ICLR 2017
Penn Treebank
Model BPC
Learning both hierarchical and temporal representation has been among the
long-standing challenges of recurrent neural networks. Multiscale recurrent
neural networks have been considered as a promising approach to resolve this
issue, yet there has been a lack of empirical evidence showing that this type
of models can actually capture the temporal dependencies by discovering the
latent hierarchical structure of the sequence. In this paper, we propose a
novel multiscale approach, called the hierarchical multiscale recurrent neural
networks, which can capture the latent hierarchical structure in the sequence
by encoding the temporal dependencies with different timescales using a novel
update mechanism. We show some evidence that our proposed multiscale
architecture can discover underlying hierarchical structure in the sequences
without using explicit boundary information. We evaluate our proposed model on
character-level language modelling and handwriting sequence modelling.
Junyoung Chung Sungjin Ahn Yoshua Bengio