t>0:5
0otherwise;(9)
or sample from a Bernoulli distribution z`
tBernoulli (~z`
t). Although this binary decision is a key to
our model, it is usually difﬁcult to use stochastic gradient descent to train such model with discrete
decisions as it is not differentiable.
3.3 C OMPUTING GRADIENT OF BOUNDARY DETECTOR
Training neural networks with discrete variables requires more efforts since the standard backpropagation is no longer applicable due to the non-differentiability. Among a few methods for training a
neural network with discrete variables such as the REINFORCE (Williams, 1992; Mnih & Gregor,
2014) and the straight-through estimator (Hinton, 2012; Bengio et al., 2013), we use the straightthrough estimator to train our model. The straight-through estimator is a biased estimator because the
non-differentiable function used in the forward pass (i.e., the step function in our case) is replaced by
a differentiable function during the backward pass (i.e., the hard sigmoid function in our case). The
straight-through estimator, however, is much simpler and often works more efﬁciently in practice
Learning both hierarchical and temporal representation has been among the
long-standing challenges of recurrent neural networks. Multiscale recurrent
neural networks have been considered as a promising approach to resolve this
issue, yet there has been a lack of empirical evidence showing that this type
of models can actually capture the temporal dependencies by discovering the
latent hierarchical structure of the sequence. In this paper, we propose a
novel multiscale approach, called the hierarchical multiscale recurrent neural
networks, which can capture the latent hierarchical structure in the sequence
by encoding the temporal dependencies with different timescales using a novel
update mechanism. We show some evidence that our proposed multiscale
architecture can discover underlying hierarchical structure in the sequences
without using explicit boundary information. We evaluate our proposed model on
character-level language modelling and handwriting sequence modelling.
Junyoung Chung Sungjin Ahn Yoshua Bengio