HIERARCHICAL MULTISCALE RECURRENT NEURAL NETWORKS
Learning both hierarchical and temporal representation has been among the longstanding challenges of recurrent neural networks. Multiscale recurrent neural networks have been considered as a promising approach to resolve this issue, yet there has been a lack of empirical evidence showing that this type of models can actually capture the temporal dependencies by discovering the latent hierarchical structure of the sequence. In this paper, we propose a novel multiscale approach, called the hierarchical multiscale recurrent neural network, that can capture the latent hierarchical structure in the sequence by encoding the temporal dependencies with different timescales using a novel update mechanism. We show some evidence that the proposed model can discover underlying hierarchical structure in the sequences without using explicit boundary information. We evaluate our proposed model on character-level language modelling and handwriting sequence generation.
1 INTRODUCTION
One of the key principles of learning in deep neural networks as well as in the human brain is to obtain a hierarchical representation with increasing levels of abstraction (Bengio, 2009; LeCun et al., 2015; Schmidhuber, 2015). A stack of representation layers, learned from the data in a way to optimize the target task, make deep neural networks entertain advantages such as generalization to unseen examples (Hoffman et al., 2013), sharing learned knowledge among multiple tasks, and discovering disentangling factors of variation (Kingma & Welling, 2013). The remarkable recent successes of the deep convolutional neural networks are particularly based on this ability to learn hierarchical representation for spatial data (Krizhevsky et al., 2012). For modelling temporal data, the recent resurgence of recurrent neural networks (RNN) has led to remarkable advances (Mikolov et al., 2010; Graves, 2013; Cho et al., 2014; Sutskever et al., 2014; Vinyals et al., 2015). However, unlike the spatial data, learning both hierarchical and temporal representation has been among the long-standing challenges of RNNs in spite of the fact that hierarchical multiscale structures naturally exist in many temporal data (Schmidhuber, 1991; Mozer, 1993; El Hihi & Bengio, 1995; Lin et al., 1996; Koutník et al., 2014).
A promising approach to model such hierarchical and temporal representation is the multiscale RNNs (Schmidhuber, 1992; El Hihi & Bengio, 1995; Koutník et al., 2014). Based on the observation that high-level abstraction changes slowly with temporal coherency while low-level abstraction has quickly changing features sensitive to the precise local timing (El Hihi & Bengio, 1995), the multiscale RNNs group hidden units into multiple modules of different timescales. In addition to the fact that the architecture fits naturally to the latent hierarchical structures in many temporal data, the multiscale approach provides the following advantages that resolve some inherent problems of standard RNNs: (a) computational efficiency obtained by updating the high-level layers less frequently, (b) efficiently delivering long-term dependencies with fewer updates at the high-level layers, which mitigates the vanishing gradient problem, (c) flexible resource allocation (e.g., more hidden units to the higher layers that focus on modelling long-term dependencies and less hidden units to the lower layers which are in charge of learning short-term dependencies). In addition, the learned latent hierarchical structures can provide useful information to other downstream tasks such ∗Yoshua Bengio is CIFAR Senior Fellow.
as module structures in computer program learning, sub-task structures in hierarchical reinforcement learning, and story segments in video understanding.
There have been various approaches to implementing the multiscale RNNs. The most popular approach is to set the timescales as hyperparameters (El Hihi & Bengio, 1995; Koutník et al., 2014; Bahdanau et al., 2016) instead of treating them as dynamic variables that can be learned from the data (Schmidhuber, 1991; 1992; Chung et al., 2015; 2016). However, considering the fact that non-stationarity is prevalent in temporal data, and that many entities of abstraction such as words and sentences are in variable length, we claim that it is important for an RNN to dynamically adapt its timescales to the particulars of the input entities of various length. While this is trivial if the hierarchical boundary structure is provided (Sordoni et al., 2015), it has been a challenge for an RNN to discover the latent hierarchical structure in temporal data without explicit boundary information.
In this paper, we propose a novel multiscale RNN model, which can learn the hierarchical multiscale structure from temporal data without explicit boundary information. This model, called a hierarchical multiscale recurrent neural network (HM-RNN), does not assign fixed update rates, but adaptively determines proper update times corresponding to different abstraction levels of the layers. We find that this model tends to learn fine timescales for low-level layers and coarse timescales for high-level layers. To do this, we introduce a binary boundary detector at each layer. The boundary detector is turned on only at the time steps where a segment of the corresponding abstraction level is completely processed. Otherwise, i.e., during the within segment processing, it stays turned off. Using the hierarchical boundary states, we implement three operations, UPDATE, COPY and FLUSH, and choose one of them at each time step. The UPDATE operation is similar to the usual update rule of the long short-term memory (LSTM) (Hochreiter & Schmidhuber, 1997), except that it is executed sparsely according to the detected boundaries. The COPY operation simply copies the cell and hidden states of the previous time step. Unlike the leaky integration of the LSTM or the Gated Recurrent Unit (GRU) (Cho et al., 2014), the COPY operation retains the whole states without any loss of information. The FLUSH operation is executed when a boundary is detected, where it first ejects the summarized representation of the current segment to the upper layer and then reinitializes the states to start processing the next segment. Learning to select a proper operation at each time step and to detect the boundaries, the HM-RNN discovers the latent hierarchical structure of the sequences. We find that the straight-through estimator (Hinton, 2012; Bengio et al., 2013; Courbariaux et al., 2016) is efficient for training this model containing discrete variables.
We evaluate our model on two tasks: character-level language modelling and handwriting sequence generation. For the character-level language modelling, the HM-RNN achieves the state-of-the-art results on the Text8 dataset, and comparable results to the state-of-the-art on the Penn Treebank and Hutter Prize Wikipedia datasets. The HM-RNN also outperforms the standard RNN on the handwriting sequence generation using the IAM-OnDB dataset. In addition, we demonstrate that the hierarchical structure found by the HM-RNN is indeed very similar to the intrinsic structure observed in the data. The contributions of this paper are:
• We propose for the first time an RNN model that can learn a latent hierarchical structure of a sequence without using explicit boundary information. • We show that it is beneficial to utilize the above structure through empirical evaluation. • We show that the straight-through estimator is an efficient way of training a model containing
discrete variables. • We propose the slope annealing trick to improve the training procedure based on the
straight-through estimator.
2 RELATED WORK
Two notable early attempts inspiring our model are Schmidhuber (1992) and El Hihi & Bengio (1995). In these works, it is advocated to stack multiple layers of RNNs in a decreasing order of update frequency for computational and learning efficiency. In Schmidhuber (1992), the author shows a model that can self-organize a hierarchical multiscale structure. Particularly in El Hihi & Bengio (1995), the advantages of incorporating a priori knowledge, “temporal dependencies are structured hierarchically", into the RNN architecture is studied. The authors propose an RNN architecture that updates each layer with a fixed but different rate, called a hierarchical RNN.
LSTMs (Hochreiter & Schmidhuber, 1997) employ the multiscale update concept, where the hidden units have different forget and update rates and thus can operate with different timescales. However, unlike our model, these timescales are not organized hierarchically. Although the LSTM has a selfloop for the gradients that helps to capture the long-term dependencies by mitigating the vanishing gradient problem, in practice, it is still limited to a few hundred time steps due to the leaky integration by which the contents to memorize for a long-term is gradually diluted at every time step. Also, the model remains computationally expensive because it has to perform the update at every time step for each unit. However, our model is less prone to these problems because it learns a hierarchical structure such that, by design, high-level layers learn to perform less frequent updates than low-level layers. We hypothesize that this property mitigates the vanishing gradient problem more efficiently while also being computationally more efficient.
A more recent model, the clockwork RNN (CW-RNN) (Koutník et al., 2014) extends the hierarchical RNN (El Hihi & Bengio, 1995) and the NARX RNN (Lin et al., 1996)1. The CW-RNN tries to solve the issue of using soft timescales in the LSTM, by explicitly assigning hard timescales. In the CW-RNN, hidden units are partitioned into several modules, and different timescales are assigned to the modules such that a module i updates its hidden units at every 2(i−1)-th time step. The CW-RNN is computationally more efficient than the standard RNN including the LSTM since hidden units are updated only at the assigned clock rates. However, finding proper timescales in the CW-RNN remains as a challenge whereas our model learns the intrinsic timescales from the data. In the biscale RNNs (Chung et al., 2016), the authors proposed to model layer-wise timescales adaptively by having additional gating units, however this approach still relies on the soft gating mechanism like LSTMs.
Other forms of Hierarchical RNN (HRNN) architectures have been proposed in the cases where the explicit hierarchical boundary structure is provided. In Ling et al. (2015), after obtaining the word boundary via tokenization, the HRNN architecture is used for neural machine translation by modelling the characters and words using the first and second RNN layers, respectively. A similar HRNN architecture is also adopted in Sordoni et al. (2015) to model dialogue utterances. However, in many cases, hierarchical boundary information is not explicitly observed or expensive to obtain. Also, it is unclear how to deploy more layers than the number of boundary levels that is explicitly observed in the data.
While the above models focus on online prediction problems, where a prediction needs to be made by using only the past data, in some cases, predictions are made after observing the whole sequence. In this setting, the input sequence can be regarded as 1-D spatial data, convolutional neural networks with 1-D kernels are proposed in Kim (2014) and Kim et al. (2015) for language modelling and sentence classification. Also, in Chan et al. (2016) and Bahdanau et al. (2016), the authors proposed to obtain high-level representation of the sequences of reduced length by repeatedly merging or pooling the lower-level representation of the sequences.
Hierarchical RNN architectures have also been used to discover the segmentation structure in sequences (Fernández et al., 2007; Kong et al., 2015). It is however different to our model in the sense that they optimize the objective with explicit labels on the hierarchical segments while our model discovers the intrinsic structure only from the sequences without segment label information.
The COPY operation used in our model can be related to Zoneout (Krueger et al., 2016) which is a recurrent generalization of stochastic depth (Huang et al., 2016). In Zoneout, an identity transformation is randomly applied to each hidden unit at each time step according to a Bernoulli distribution. This results in occasional copy operations of the previous hidden states. While the focus of Zoneout is to propose a regularization technique similar to dropout (Srivastava et al., 2014) (where the regularization strength is controlled by a hyperparameter), our model learns (a) to dynamically determine when to copy from the context inputs and (b) to discover the hierarchical multiscale structure and representation. Although the main goal of our proposed model is not regularization, we found that our model also shows very good generalization performance.
3 HIERARCHICAL MULTISCALE RECURRENT NEURAL NETWORKS
3.1 MOTIVATION
To begin with, we provide an example of how a stacked RNN can model temporal data in an ideal setting, i.e., when the hierarchy of segments is provided (Sordoni et al., 2015; Ling et al., 2015). In Figure 1 (a), we depict a hierarchical RNN (HRNN) for language modelling with two layers: the first layer receives characters as inputs and generates word-level representations (C2W-RNN), and the second layer takes the word-level representations as inputs and yields phrase-level representations (W2P-RNN).
As shown, by means of the provided end-of-word labels, the C2W-RNN obtains word-level representation after processing the last character of each word and passes the word-level representation to the W2P-RNN. Then, the W2P-RNN performs an update of the phrase-level representation. Note that the hidden states of the W2P-RNN remains unchanged while all the characters of a word are processed by the C2W-RNN. When the C2W-RNN starts to process the next word, its hidden states are reinitialized using the latest hidden states of the W2P-RNN, which contain summarized representation of all the words that have been processed by that time step, in that phrase.
From this simple example, we can see the advantages of having a hierarchical multiscale structure: (1) as the W2P-RNN is updated at a much slower update rate than the C2W-RNN, a considerable amount of computation can be saved, (2) gradients are backpropagated through a much smaller number of time steps, and (3) layer-wise capacity control becomes possible (e.g., use a smaller number of hidden units in the first layer which models short-term dependencies but whose updates are invoked much more often).
Can an RNN discover such hierarchical multiscale structure without explicit hierarchical boundary information? Considering the fact that the boundary information is difficult to obtain (for example, consider languages where words are not always cleanly separated by spaces or punctuation symbols, and imperfect rules are used to separately perform segmentation) or usually not provided at all, this is a legitimate problem. It gets worse when we consider higher-level concepts which we would like the RNN to discover autonomously. In Section 2, we discussed the limitations of the existing RNN models under this setting, which either have to update all units at every time step or use fixed update frequencies (El Hihi & Bengio, 1995; Koutník et al., 2014). Unfortunately, this kind of approach is not well suited to the case where different segments in the hierarchical decomposition have different lengths: for example, different words have different lengths, so a fixed hierarchy would not update its upper-level units in synchrony with the natural boundaries in the data.
3.2 THE PROPOSED MODEL
A key element of our model is the introduction of a parametrized boundary detector, which outputs a binary value, in each layer of a stacked RNN, and learns when a segment should end in such a way to optimize the overall target objective. Whenever the boundary detector is turned on at a time step of layer ` (i.e., when the boundary state is 1), the model considers this to be the end of a
1The acronym NARX stands for Non-linear Auto-Regressive model with eXogenous inputs.
segment corresponding to the latent abstraction level of that layer (e.g., word or phrase) and feeds the summarized representation of the detected segment into the upper layer (`+ 1). Using the boundary states, at each time step, each layer selects one of the following operations: UPDATE, COPY or FLUSH. The selection is determined by (1) the boundary state of the current time step in the layer below z`−1t and (2) the boundary state of the previous time step in the same layer z ` t−1.
In the following, we describe an HM-RNN based on the LSTM update rule. We call this model a hierarchical multiscale LSTM (HM-LSTM). Consider an HM-LSTM model of L layers (` = 1, . . . , L) which, at each layer `, performs the following update at time step t:
h`t, c ` t, z ` t = f ` HM-LSTM(c ` t−1,h ` t−1,h `−1 t ,h `+1 t−1, z ` t−1, z `−1 t ). (1)
Here, h and c denote the hidden and cell states, respectively. The function f `HM-LSTM is implemented as follows. First, using the two boundary states z`t−1 and z `−1 t , the cell state is updated by:
c`t =  f `t c`t−1 + i`t g`t if z`t−1 = 0 and z`−1t = 1 (UPDATE) c`t−1 if z ` t−1 = 0 and z `−1 t = 0 (COPY)
i`t g`t if z`t−1 = 1 (FLUSH), (2)
and then the hidden state is obtained by:
h`t = { h`t−1 if COPY, o`t tanh(c`t) otherwise.
(3)
Here, (f , i,o) are forget, input, output gates, and g is a cell proposal vector. Note that unlike the LSTM, it is not necessary to compute these gates and cell proposal values at every time step. For example, in the case of the COPY operation, we do not need to compute any of these values and thus can save computations.
The COPY operation, which simply performs (c`t,h ` t)← (c`t−1,h`t−1), implements the observation that an upper layer should keep its state unchanged until it receives the summarized input from the lower layer. The UPDATE operation is performed to update the summary representation of the layer ` if the boundary z`−1t is detected from the layer below but the boundary z ` t−1 was not found at the previous time step. Hence, the UPDATE operation is executed sparsely unlike the standard RNNs where it is executed at every time step, making it computationally inefficient. If a boundary is detected, the FLUSH operation is executed. The FLUSH operation consists of two sub-operations: (a) EJECT to pass the current state to the upper layer and then (b) RESET to reinitialize the state before starting to read a new segment. This operation implicitly forces the upper layer to absorb the summary information of the lower layer segment, because otherwise it will be lost. Note that the FLUSH operation is a hard reset in the sense that it completely erases all the previous states of the same layer, which is different from the soft reset or soft forget operation in the GRU or LSTM.
Whenever needed (depending on the chosen operation), the gate values (f `t , i ` t,o ` t), the cell proposal g`t , and the pre-activation of the boundary detector z̃ ` t
2 are then obtained by: f `t i`t o`t g`t z̃`t  =  sigm sigm sigm tanh hard sigm  fslice (srecurrent(`)t + stop-down(`)t + sbottom-up(`)t + b(`)) , (4) where
s recurrent(`) t = U ` `h ` t−1, (5)
s top-down(`) t = z ` t−1U ` `+1h `+1 t−1, (6)
s bottom-up(`) t = z `−1 t W ` `−1h `−1 t . (7)
Here, we useW ji ∈ R(4dim(h `)+1)×dim(h`−1), U ji ∈ R(4dim(h `)+1)×dim(h`) to denote state transition parameters from layer i to layer j, and b ∈ R4dim(h`)+1 is a bias term. In the last layer L, the
2z̃`t can also be implemented as a function of h`t , e.g., z̃`t = hard sigm(Uh`t).
top-down connection is ignored, and we use h0t = xt. Since the input should not be omitted, we set z0t = 1 for all t. Also, we do not use the boundary detector for the last layer. The hard sigm is defined by hard sigm(x) = max ( 0,min ( 1, ax+1
2
)) with a being the slope variable.
Unlike the standard LSTM, the HM-LSTM has a top-down connection from (`+ 1) to `, which is allowed to be activated only if a boundary is detected at the previous time step of the layer ` (see Eq. 6). This makes the layer ` to be initialized with more long-term information after the boundary is detected and execute the FLUSH operation. In addition, the input from the lower layer (` − 1) becomes effective only when a boundary is detected at the current time step in the layer (`− 1) due to the binary gate z`−1t . Figure 2 (left) shows the gating mechanism of the HM-LSTM at time step t.
Finally, the binary boundary state z`t is obtained by:
z`t = fbound(z̃ ` t ). (8)
For the binarization function fbound : R→ {0, 1}, we can either use a deterministic step function:
z`t = { 1 if z̃`t > 0.5 0 otherwise,
(9)
or sample from a Bernoulli distribution z`t ∼ Bernoulli(z̃`t ). Although this binary decision is a key to our model, it is usually difficult to use stochastic gradient descent to train such model with discrete decisions as it is not differentiable.
3.3 COMPUTING GRADIENT OF BOUNDARY DETECTOR
Training neural networks with discrete variables requires more efforts since the standard backpropagation is no longer applicable due to the non-differentiability. Among a few methods for training a neural network with discrete variables such as the REINFORCE (Williams, 1992; Mnih & Gregor, 2014) and the straight-through estimator (Hinton, 2012; Bengio et al., 2013), we use the straightthrough estimator to train our model. The straight-through estimator is a biased estimator because the non-differentiable function used in the forward pass (i.e., the step function in our case) is replaced by a differentiable function during the backward pass (i.e., the hard sigmoid function in our case). The straight-through estimator, however, is much simpler and often works more efficiently in practice than other unbiased but high-variance estimators such as the REINFORCE. The straight-through estimator has also been used in Courbariaux et al. (2016) and Vezhnevets et al. (2016).
The Slope Annealing Trick. In our experiment, we use the slope annealing trick to reduce the bias of the straight-through estimator. The idea is to reduce the discrepancy between the two functions used during the forward pass and the backward pass. That is, by gradually increasing the slope a of the hard sigmoid function, we make the hard sigmoid be close to the step function. Note that starting with a high slope value from the beginning can make the training difficult while it is more applicable later when the model parameters become more stable. In our experiments, starting from slope a = 1, we slowly increase the slope until it reaches a threshold with an appropriate scheduling.
4 EXPERIMENTS
We evaluate the proposed model on two tasks, character-level language modelling and handwriting sequence generation. Character-level language modelling is a representative example of discrete
sequence modelling, where the discrete symbols form a distinct hierarchical multiscale structure. The performance on real-valued sequences is tested on the handwriting sequence generation in which a relatively clear hierarchical multiscale structure exists compared to other data such as speech signals.
4.1 CHARACTER-LEVEL LANGUAGE MODELLING
A sequence modelling task aims at learning the probability distribution over sequences by minimizing the negative log-likelihood of the training sequences:
min θ − 1 N N∑ n=1 Tn∑ t=1 log p (xnt | xn<t; θ) , (10)
where θ is the model parameter, N is the number of training sequences, and Tn is the length of the n-th sequence. A symbol at time t of sequence n is denoted by xnt , and x n <t denotes all previous symbols at time t. We evaluate our model on three benchmark text corpora: (1) Penn Treebank, (2) Text8 and (3) Hutter Prize Wikipedia. We use the bits-per-character (BPC), E[− log2 p(xt+1 | x≤t)], as the evaluation metric.
Model We use a model consisting of an input embedding layer, an RNN module and an output module. The input embedding layer maps each input symbol into 128-dimensional continuous vector without using any non-linearity. The RNN module is the HM-LSTM, described in Section 3, with three layers. The output module is a feedforward neural network with two layers, an output embedding layer and a softmax layer. Figure 2 (right) shows a diagram of the output module. At each time step, the output embedding layer receives the hidden states of the three RNN layers as input. In order to adaptively control the importance of each layer at each time step, we also introduce three scalar gating units g`t ∈ R to each of the layer outputs:
g`t = sigm(w `[h1t ; · · · ;hLt ]), (11)
where w` ∈ R ∑L
`=1 dim(h `) is the weight parameter. The output embedding het is computed by:
het = ReLU ( L∑ `=1 g`tW e `h ` t ) , (12)
where L = 3 and ReLU(x) = max(0, x) (Nair & Hinton, 2010). Finally, the probability distribution for the next target character is computed by the softmax function, softmax(xj) = e
xj∑K k=1 exk , where
each output class is a character.
Penn Treebank We process the Penn Treebank dataset (Marcus et al., 1993) by following the procedure introduced in Mikolov et al. (2012). Each update is done by using a mini-batch of 64 examples of length 100 to prevent the memory overflow problem when unfolding the RNN in time for backpropagation. The last hidden state of a sequence is used to initialize the hidden state of the next sequence to approximate the full backpropagation. We train the model using Adam (Kingma & Ba, 2014) with an initial learning rate of 0.002. We divide the learning rate by a factor of 50 when the validation negative log-likelihood stopped decreasing. The norm of the gradient is clipped with a threshold of 1 (Mikolov et al., 2010; Pascanu et al., 2012). We also apply layer normalization (Ba et al., 2016) to our models. For all of the character-level language modelling experiments, we apply the same procedure, but only change the number of hidden units, mini-batch size and the initial learning rate.
For the Penn Treebank dataset, we use 512 units in each layer of the HM-LSTM and for the output embedding layer. In Table 1 (left), we compare the test BPCs of four variants of our model to other baseline models. Note that the HM-LSTM using the step function for the hard boundary decision outperforms the others using either sampling or soft boundary decision (i.e., hard sigmoid). The test BPC is further improved with the slope annealing trick, which reduces the bias of the straight-through estimator. We increased the slope a with the following schedule a = min (5, 1 + 0.04 ·Nepoch), where Nepoch is the maximum number of epochs. The HM-LSTM achieves test BPC score of 1.24. For the remaining tasks, we fixed the hard boundary decision using the step function without slope annealing due to the difficulty of finding a good annealing schedule on large-scale datasets.
Text8 The Text8 dataset (Mahoney, 2009) consists of 100M characters extracted from the Wikipedia corpus. Text8 contains only alphabets and spaces, and thus we have total 27 symbols. In order to compare with other previous works, we follow the data splits used in Mikolov et al. (2012). We use 1024 units for each HM-LSTM layer and 2048 units for the output embedding layer. The mini-batch size and the initial learning rate are set to 128 and 0.001, respectively. The results are shown in Table 2. The HM-LSTM obtains the state-of-the-art test BPC 1.29.
Hutter Prize Wikipedia The Hutter Prize Wikipedia (enwik8) dataset (Hutter, 2012) contains 205 symbols including XML markups and special characters. We follow the data splits used in Graves (2013) where the first 90M characters are used to train the model, the next 5M characters for validation, and the remainders for the test set. We use the same model size, mini-batch size and the initial learning rate as in the Text8. In Table 1 (right), we show the HM-LSTM achieving the test BPC 1.32, which is a tie with the state-of-the-art result among the neural models. Although the neural models, show remarkable performances, their compression performance is still behind the best models such as PAQ8hp12 (Mahoney, 2005) and decomp8 (Mahoney, 2009).
Visualizing Learned Hierarchical Multiscale Structure In Figure 3 and 4, we visualize the boundaries detected by the boundary detectors of the HM-LSTM while reading a character sequence of total length 270 taken from the validation set of either the Penn Treebank or Hutter Prize Wikipedia dataset. Due to the page width limit, the figure contains the sequence partitioned into three segments of length 90. The white blocks indicate boundaries z`t = 1 while the black blocks indicate the non-boundaries z`t = 0.
Interestingly in both figures, we can observe that the boundary detector of the first layer, z1, tends to be turned on when it sees a space or after it sees a space, which is a reasonable breakpoint to separate between words. This is somewhat surprising because the model self-organizes this structure
without any explicit boundary information. In Figure 3, we observe that the z1 tends to detect the boundaries of the words but also fires within the words, where the z2 tends to fire when it sees either an end of a word or 2, 3-grams. In Figure 4, we also see flushing in the middle of a word, e.g., “tele-FLUSH-phone”. Note that “tele” is a prefix after which a various number of postfixes can follow. From these, it seems that the model uses to some extent the concept of surprise to learn the boundary. Although interpretation of the second layer boundaries is not as apparent as the first layer boundaries, it seems to segment at reasonable semantic / syntactic boundaries, e.g., “consumers may” - “want to move their telephones a” - “little closer to the tv set <unk>”, and so on.
Another remarkable point is the fact that we do not pose any constraint on the number of boundaries that the model can fire up. The model, however, learns that it is more beneficial to delay the information ejection to some extent. This is somewhat counterintuitive because it might look more beneficial to feed the fresh update to the upper layers at every time step without any delay. We conjecture the reason that the model works in this way is due to the FLUSH operation that poses an implicit constraint on the frequency of boundary detection, because it contains both a reward (feeding fresh information to upper layers) and a penalty (erasing accumulated information). The model finds an optimal balance between the reward and the penalty.
To understand the update mechanism more intuitively, in Figure 4, we also depict the heatmap of the `2-norm of the hidden states along with the states of the boundary detectors. As we expect, we can see that there is no change in the norm value within segments due to the COPY operation. Also, the color of ‖h1‖ changes quickly (at every time step) because there is no COPY operation in the first layer. The color of ‖h2‖ changes less frequently based on the states of z1t and z2t−1. The color of ‖h3‖ changes even slowly, i.e., only when z2t = 1. A notable advantage of the proposed architecture is that the internal process of the RNN becomes more interpretable. For example, we can substitute the states of z1t and z 2 t−1 into Eq. 2 and infer which operation among the UPDATE, COPY and FLUSH was applied to the second layer at time step t. We can also inspect the update frequencies of the layers simply by counting how many UPDATE and FLUSH operations were made in each layer. For example in Figure 4, we see that the first layer updates at every time step (which is 270 UPDATE operations), the second layer updates 56 times,
and only 9 updates has made in the third layer. Note that, by design, the first layer performs UPDATE operation at every time step and then the number of UPDATE operations decreases as the layer level increases. In this example, the total number of updates is 335 for the HM-LSTM which is 60% of reduction from the 810 updates of the standard RNN architecture.
4.2 HANDWRITING SEQUENCE GENERATION
We extend the evaluation of the HM-LSTM to a real-valued sequence modelling task using IAMOnDB (Liwicki & Bunke, 2005) dataset. The IAM-OnDB dataset consists of 12, 179 handwriting examples, each of which is a sequence of (x, y) coordinate and a binary indicator p for pen-tip location, giving us (x1:Tn , y1:Tn , p1:Tn), where n is an index of a sequence. At each time step, the model receives (xt, yt, pt), and the goal is to predict (xt+1, yt+1, pt+1). The pen-up (pt = 1) indicates an end of a stroke, and the pen-down (pt = 0) indicates that a stroke is in progress. There is usually a large shift in the (x, y) coordinate to start a new stroke after the pen-up happens. We remove all sequences whose length is shorter than 300. This leaves us 10, 465 sequences for training, 581 for validation, 582 for test. The average length of the sequences is 648. We normalize the range of the (x, y) coordinates separately with the mean and standard deviation obtained from the training set. We use the mini-batch size of 32, and the initial learning rate is set to 0.0003.
We use the same model architecture as used in the character-level language model, except that the output layer is modified to predict real-valued outputs. We use the mixture density network as the output layer following Graves (2013), and use 400 units for each HM-LSTM layer and for the output embedding layer. In Table 3, we compare the log-likelihood averaged over the test sequences of the IAM-OnDB dataset. We observe that the HM-LSTM outperforms the standard LSTM. The slope annealing trick further improves the test log-likelihood of the HM-LSTM into 1167 in our setting. In this experiment, we increased the slope a with the following schedule a = min (3, 1 + 0.004 ·Nepoch). In Figure 5, we let the HM-LSTM to read a randomly picked validation sequence and present the visualization of handwriting examples by segments based on either the states of z2 or the states of pen-tip location3.
5 CONCLUSION
In this paper, we proposed the HM-RNN that can capture the latent hierarchical structure of the sequences. We introduced three types of operations to the RNN, which are the COPY, UPDATE and FLUSH operations. In order to implement these operations, we introduced a set of binary variables and a novel update rule that is dependent on the states of these binary variables. Each binary variable is learned to find segments at its level, therefore, we call this binary variable, a boundary detector. On the character-level language modelling, the HM-LSTM achieved state-of-the-art result on the Text8 dataset and comparable results to the state-of-the-art results on the Penn Treebank and Hutter Prize Wikipedia datasets. Also, the HM-LSTM outperformed the standard LSTM on the handwriting sequence generation. Our results and analysis suggest that the proposed HM-RNN can discover the latent hierarchical structure of the sequences and can learn efficient hierarchical multiscale representation that leads to better generalization performance.
3The plot function could be found at blog.otoro.net/2015/12/12/handwriting-generation-demo-in-tensorflow/.
ACKNOWLEDGMENTS
The authors would like to thank Alex Graves, Tom Schaul and Hado van Hasselt for their fruitful comments and discussion. We acknowledge the support of the following agencies for research funding and computing support: Ubisoft, Samsung, IBM, Facebook, Google, Microsoft, NSERC, Calcul Québec, Compute Canada, the Canada Research Chairs and CIFAR. The authors thank the developers of Theano (Team et al., 2016). JC would like to thank Arnaud Bergenon and Frédéric Bastien for their technical support. JC would also like to thank Guillaume Alain, Kyle Kastner and David Ha for providing us useful pieces of code.

This paper proposes a novel variant of recurrent networks that is able to learn the hierarchy of information in sequential data (e.g., character->word). Their approach does not require boundary information to segment the sequence in meaningful groups (like in Chung et al., 2016).
Their model is organized as a set of layers that aim at capturing the information form different “level of abstraction”. The lowest level activate the upper one and decide when to update it based on a controller (or state cell, called c). A key feature of their model is that c is a discrete variable, allowing potentially fast inference time. However, this makes their model more challenging to learn, leading to the use of the straight-through estimator by Hinton, 2012.
The experiment section is thorough and their model obtain competitive performance on several challenging tasks. The qualitative results show also that their model can capture natural boundaries.
Overall this paper presents a strong and novel model with promising experimental results.
On a minor note, I have few remarks/complaints about the writing and the related work:
- In the introduction:
“One of the key principles of learning in deep neural networks as well as in the human brain” : please provide evidence for the “human brain” part of this claim.
“For modelling temporal data, the recent resurgence of recurrent neural networks (RNN) has led to remarkable advances” I believe you re missing Mikolov et al. 2010 in the references.
“in spite of the fact that hierarchical multiscale structures naturally exist in many temporal data”: missing reference to Lin et al., 1996
- in the related work:
“A more recent model, the clockwork RNN (CW-RNN) (Koutník et al., 2014) extends the hierarchicalRNN (El Hihi & Bengio, 1995)” : It extends the NARX model of Lin et al. 1996, not the El Hihi & Bengio, 1995.
While the above models focus on online prediction problems, where a prediction needs to be made…”: I believe there is a lot of missing references, in particular to Socher’s work or older recursive networks.
“The norm of the gradient is clipped with a threshold of 1 (Pascanu et al., 2012)”: this is not the first work using gradient clipping. I believe it was introduced in Mikolov et al., 2010.
Missing references:
“Recurrent neural network based language model.”, Mikolov et al. 2010
“Learning long-term dependencies in NARX recurrent neural networks”, Lin et al. 1996
“Sequence labelling in structured domains with hierarchical recurrent neural networks“, Fernandez et al. 2007
“Learning sequential tasks by incrementally adding  higher  orders”, Ring, 1993


OTHER MODIFICATIONS
PixelCNNs are a recently proposed class of powerful generative models with tractable likelihood. Here we discuss our implementation of PixelCNNs which we make available at https://github.com/openai/pixel-cnn. Our implementation contains a number of modifications to the original model that both simplify its structure and improve its performance. 1) We use a discretized logistic mixture likelihood on the pixels, rather than a 256-way softmax, which we find to speed up training. 2) We condition on whole pixels, rather than R/G/B sub-pixels, simplifying the model structure. 3) We use downsampling to efficiently capture structure at multiple resolutions. 4) We introduce additional short-cut connections to further speed up optimization. 5) We regularize the model using dropout. Finally, we present state-of-the-art log likelihood results on CIFAR-10 to demonstrate the usefulness of these modifications.
1 INTRODUCTION
The PixelCNN, introduced by van den Oord et al. (2016b), is a generative model of images with a tractable likelihood. The model fully factorizes the probability density function on an image x over all its sub-pixels (color channels in a pixel) as p(x) = ∏ i p(xi|x<i). The conditional distributions p(xi|x<i) are parameterized by convolutional neural networks and all share parameters. The PixelCNN is a powerful model as the functional form of these conditionals is very flexible. In addition it is computationally efficient as all conditionals can be evaluated in parallel on a GPU for an observed image x. Thanks to these properties, the PixelCNN represents the current state-of-the-art in generative modeling when evaluated in terms of log-likelihood. Besides being used for modeling images, the PixelCNN model was recently extended to model audio (van den Oord et al., 2016a), video (Kalchbrenner et al., 2016b) and text (Kalchbrenner et al., 2016a).
For use in our research, we developed our own internal implementation of PixelCNN and made a number of modifications to the base model to simplify its structure and improve its performance. We now release our implementation at https://github.com/openai/pixel-cnn, hoping that it will be useful to the broader community. Our modifications are discussed in Section 2, and evaluated experimentally in Section 3. State-of-the-art log-likelihood results confirm their usefulness.
2 MODIFICATIONS TO PIXELCNN
We now describe the most important modifications we have made to the PixelCNN model architecure as described by van den Oord et al. (2016c). For complete details see our code release at https://github.com/openai/pixel-cnn.
2.1 DISCRETIZED LOGISTIC MIXTURE LIKELIHOOD
The standard PixelCNN model specifies the conditional distribution of a sub-pixel, or color channel of a pixel, as a full 256-way softmax. This gives the model a lot of flexibility, but it is also very costly in terms of memory. Moreover, it can make the gradients with respect to the network parameters
very sparse, especially early in training. With the standard parameterization, the model does not know that a value of 128 is close to a value of 127 or 129, and this relationship first has to be learned before the model can move on to higher level structures. In the extreme case where a particular sub-pixel value is never observed, the model will learn to assign it zero probability. This would be especially problematic for data with higher accuracy on the observed pixels than the usual 8 bits: In the extreme case where very high precision values are observed, the PixelCNN, in its current form, would require a prohibitive amount of memory and computation, while learning very slowly. We therefore propose a different mechanism for computing the conditional probability of the observed discretized pixel values. In our model, like in the VAE of Kingma et al. (2016), we assume there is a latent color intensity ν with a continuous distribution, which is then rounded to its nearest 8-bit representation to give the observed sub-pixel value x. By choosing a simple continuous distribution for modeling ν (like the logistic distribution as done by Kingma et al. (2016)) we obtain a smooth and memory efficient predictive distribution for x. Here, we take this continuous univariate distribution to be a mixture of logistic distributions which allows us to easily calculate the probability on the observed discretized value x, as shown in equation (2). For all sub-pixel values x excepting the edge cases 0 and 255 we have:
ν ∼ K∑ i=1 πilogistic(µi, si) (1)
P (x|π, µ, s) = K∑ i=1 πi [σ((x+ 0.5− µi)/si)− σ((x− 0.5− µi)/si)] , (2)
where σ() is the logistic sigmoid function. For the edge case of 0, replace x − 0.5 by −∞, and for 255 replace x + 0.5 by +∞. Our provided code contains a numerically stable implementation for calculating the log of the probability in equation 2.
Our approach follows earlier work using continuous mixture models (Domke et al., 2008; Theis et al., 2012; Uria et al., 2013; Theis & Bethge, 2015), but avoids allocating probability mass to values outside the valid range of [0, 255] by explicitly modeling the rounding of ν to x. In addition, we naturally assign higher probability to the edge values 0 and 255 than to their neighboring values, which corresponds well with the observed data distribution as shown in Figure 1. Experimentally, we find that only a relatively small number of mixture components, say 5, is needed to accurately model the conditional distributions of the pixels. The output of our network is thus of much lower dimension, yielding much denser gradients of the loss with respect to our parameters. In our experiments this greatly sped up convergence during optimization, especially early on in training. However, due to the other changes in our architecture compared to that of van den Oord et al. (2016c) we cannot say with certainty that this would also apply to the original PixelCNN model.
2.2 CONDITIONING ON WHOLE PIXELS
The pixels in a color image consist of three real numbers, giving the intensities of the red, blue and green colors. The original PixelCNN factorizes the generative model over these 3 sub-pixels. This allows for very general dependency structure, but it also complicates the model: besides keeping track of the spatial location of feature maps, we now have to separate out all feature maps in 3 groups depending on whether or not they can see the R/G/B sub-pixel of the current location. This added complexity seems to be unnecessary as the dependencies between the color channels of a pixel are likely to be relatively simple and do not require a deep network to model. Therefore, we instead condition only on whole pixels up and to the left in an image, and output joint predictive distributions over all 3 channels of a predicted pixel. The predictive distribution on a pixel itself can be interpreted as a simple factorized model: We first predict the red channel using a discretized mixture of logistics as described in section 2.1. Next, we predict the green channel using a predictive distribution of the same form. Here we allow the means of the mixture components to linearly depend on the value of the red sub-pixel. Finally, we model the blue channel in the same way, where we again only allow linear dependency on the red and green channels. For the pixel (ri,j , gi,j , bi,j) at location (i, j) in our image, the distribution conditional on the context Ci,j , consisting of the mixture indicator and the previous pixels, is thus
p(ri,j , gi,j , bi,j |Ci,j) = P (ri,j |µr(Ci,j), sr(Ci,j))× P (gi,j |µg(Ci,j , ri,j), sg(Ci,j)) ×P (bi,j |µb(Ci,j , ri,j , gi,j), sb(Ci,j))
µg(Ci,j , ri,j) = µg(Ci,j) + α(Ci,j)ri,j
µb(Ci,j , ri,j , gi,j) = µb(Ci,j) + β(Ci,j)ri,j + γ(Ci,j)bi,j , (3)
with α, β, γ scalar coefficients depending on the mixture component and previous pixels.
The mixture indicator is shared across all 3 channels; i.e. our generative model first samples a mixture indicator for a pixel, and then samples the color channels one-by-one from the corresponding mixture component. Had we used a discretized mixture of univariate Gaussians for the sub-pixels, instead of logistics, this would have been exactly equivalent to predicting the complete pixel using a (discretized) mixture of 3-dimensional Gaussians with full covariance. The logistic and Gaussian distributions are very similar, so this is indeed very close to what we end up doing. For full implementation details we refer to our code at https://github.com/openai/pixel-cnn.
2.3 DOWNSAMPLING VERSUS DILATED CONVOLUTION
The original PixelCNN only uses convolutions with small receptive field. Such convolutions are good at capturing local dependencies, but not necessarily at modeling long range structure. Although we find that capturing these short range dependencies is often enough for obtaining very good log-likelihood scores (see Table 2), explicitly encouraging the model to capture long range dependencies can improve the perceptual quality of generated images (compare Figure 3 and Figure 5). One way of allowing the network to model structure at multiple resolutions is to introduce dilated convolutions into the model, as proposed by van den Oord et al. (2016a) and Kalchbrenner et al. (2016b). Here, we instead propose to use downsampling by using convolutions of stride 2. Downsampling accomplishes the same multi-resolution processing afforded by dilated convolutions, but at a reduced computational cost: where dilated convolutions operate on input of ever increasing size (due to zero padding), downsampling reduces the input size by a factor of 4 (for stride of 2 in 2 dimensions) at every downsampling. The downside of using downsampling is that it loses information, but we can compensate for this by introducing additional short-cut connections into the network as explained in the next section. With these additional short-cut connections, we found the performance of downsampling to be the same as for dilated convolution.
2.4 ADDING SHORT-CUT CONNECTIONS
For input of size 32 × 32 our suggested model consists of 6 blocks of 5 ResNet layers. In between the first and second block, as well as the second and third block, we perform subsampling by strided convolution. In between the fourth and fifth block, as well as the fifth and sixth block, we perform upsampling by transposed strided convolution. This subsampling and upsampling process loses information, and we therefore introduce additional short-cut connections into the model to recover
this information from lower layers in the model. The short-cut connections run from the ResNet layers in the first block to the corresponding layers in the sixth block, and similarly between blocks two and five, and blocks three and four. This structure resembles the VAE model with top down inference used by Kingma et al. (2016), as well as the U-net used by Ronneberger et al. (2015) for image segmentation. Figure 2 shows our model structure graphically.
2.5 REGULARIZATION USING DROPOUT
The PixelCNN model is powerful enough to overfit on training data. Moreover, rather than just reproducing the training images, we find that overfitted models generate images of low perceptual quality, as shown in Figure 8. One effective way of regularizing neural networks is dropout (Srivastava et al., 2014). For our model, we apply standard binary dropout on the residual path after the first convolution. This is similar to how dropout is applied in the wide residual networks of Zagoruyko & Komodakis (2016). Using dropout allows us to successfully train high capacity models while avoiding overfitting and producing high quality generations (compare figure 8 and figure 3).
3 EXPERIMENTS
We apply our model to modeling natural images in the CIFAR-10 data set. We achieve state-of-theart results in terms of log-likelihood, and generate images with coherent global structure.
3.1 UNCONDITIONAL GENERATION ON CIFAR-10
We apply our PixelCNN model, with the modifications as described above, to generative modeling of the images in the CIFAR-10 data set. For the encoding part of the PixelCNN, the model uses 3 Resnet blocks consisting of 5 residual layers, with 2× 2 downsampling in between. The same architecture is used for the decoding part of the model, but with upsampling instead of downsampling in between blocks. All residual layers use 192 feature maps and a dropout rate of 0.5. Table 1 shows the stateof-the-art test log-likelihood obtained by our model. Figure 3 shows some samples generated by the model.
3.2 CLASS-CONDITIONAL GENERATION
Next, we follow van den Oord et al. (2016c) in making our generative model conditional on the class-label of the CIFAR-10 images. This is done by linearly projecting a one-hot encoding of the class-label into a separate class-dependent bias vector for each convolutional unit in our network. We find that making the model class-conditional makes it harder to avoid overfitting on the training data: our best test log-likelihood is 2.94 in this case. Figure 4 shows samples from the class-conditional model, with columns 1-10 corresponding the 10 classes in CIFAR-10. The images clearly look qualitatively different across the columns and for a number of them we can clearly identify their class label.
3.3 EXAMINING NETWORK DEPTH AND FIELD OF VIEW SIZE
It is hypothesized that the size of the receptive field and additionally the removal of blind spots in the receptive field are important for PixelCNN’s performance (van den Oord et al., 2016b). Indeed van den Oord et al. (2016c) specifically introduced an improvement over the previous PixelCNN model to remove the blind spot in the receptive field that was present in their earlier model.
Here we present the surprising finding that in fact a PixelCNN with rather small receptive field can attain competitive generative modelling performance on CIFAR-10 as long as it has enough capacity. Specifically, we experimented with our proposed PixelCNN++ model without downsampling blocks and reduce the number of layers to limit the receptive field size. We investigate two receptive field sizes: 11x5 and 15x8, and a receptive field size of 11x5, for example, means that the conditional distribution of a pixel can depends on a rectangle above the pixel of size 11x5 as well as 11−12 = 5x1 block to the left of the pixel.
As we limit the size of the receptive field, the capacity of the network also drops significantly since it contains many fewer layers than a normal PixelCNN. We call the type of PixelCNN that’s simply limited in depth “Plain” Small PixelCNN. Interestingly, this model already has better performance than the original PixelCNN in van den Oord et al. (2016b) which had a blind spot. To increase capacity, we introduced two simple variants that make Small PixelCNN more expressive without growing the receptive field:
• NIN (Network in Network): insert additional gated ResNet blocks with 1x1 convolution between regular convolution blocks that grow receptive field. In this experiment, we inserted 3 NIN blocks between every other layer. • Autoregressive Channel: skip connections between sets of channels via 1x1 convolution
gated ResNet block.
Both modifications increase the capacity of the network, resulting in improved log-likelihood as shown in Table 2. Although the model with small receptive field already achieves an impressive likelihood score, its samples do lack global structure, as seen in Figure 5.
3.4 ABLATION EXPERIMENTS
In order to test the effect of our modifications to PixelCNN, we run a number of ablation experiments where for each experiment we remove a specific modification.
3.4.1 SOFTMAX LIKELIHOOD INSTEAD OF DISCRETIZED LOGISTIC MIXTURE
In order to test the contribution of our logistic mixture likelihood, we re-run our CIFAR-10 experiment with the 256-way softmax as the output distribution instead. We allow the 256 logits for each sub-pixel to linearly depend on the observed value of previous sub-pixels, with coefficients that are given as output by the model. Our model with softmax likelihood is thus strictly more flexible than our model with logistic mixture likelihood, although the parameterization is quite different from that used by van den Oord et al. (2016c). The model now outputs 1536 numbers per pixel, describing the logits on the 256 potential values for each sub-pixel, as well as the coefficients for the dependencies between the sub-pixels. Figure 6 shows that this model trains more slowly than our original model. In addition, the running time per epoch is significantly longer for our tensorflow implementation. For our architecture, the logistic mixture model thus clearly performs better. Since our architecture differs from that of van den Oord et al. (2016c) in other ways as well, we cannot say whether this would also apply to their model.
3.4.2 CONTINUOUS MIXTURE LIKELIHOOD INSTEAD OF DISCRETIZATION
Instead of directly modeling the discrete pixel values in an image, it is also possible to de-quantize them by adding noise from the standard uniform distribution, as used by Uria et al. (2013) and others, and modeling the data as being continuous. The resulting model can be interpreted as a variational autoencoder (Kingma & Welling, 2013; Rezende et al., 2014), where the dequantized pixels z form a latent code whose prior distribution is captured by our model. Since the original discrete pixels x can be perfectly reconstructed from z under this model, the usual reconstruction term vanishes from
the variational lower bound. The entropy of the standard uniform distribution is zero, so the term that remains is the log likelihood of the dequantized pixels, which thus gives us a variational lower bound on the log likelihood of our original data.
We re-run our model for CIFAR-10 using the same model settings as those used for the 2.92 bits per dimension result in Table 1, but now we remove the discretization in our likelihood model and instead add standard uniform noise to the image data. The resulting model is a continuous mixture model in the same class as that used by Theis et al. (2012); Uria et al. (2013); Theis & Bethge (2015) and others. After optimization, this model gives a variational lower bound on the data log likelihood of 3.11 bits per dimension. The difference with the reported 2.92 bits per dimension shows the benefit of using discretization in the likelihood model.
3.4.3 NO SHORT-CUT CONNECTIONS
Next, we test the importance of the additional parallel short-cut connections in our model, indicated by the dotted lines in Figure 2. We re-run our unconditional CIFAR-10 experiment, but remove the short-cut connections from the model. As seen in Figure 7, the model fails to train without these connections. The reason for needing these extra short-cuts is likely to be our use of sub-sampling, which discards information that otherwise cannot easily be recovered,
3.4.4 NO DROPOUT
We re-run our CIFAR-10 model without dropout regularization. The log-likelihood we achieve on the training set is below 2.0 bits per sub-pixel, but the final test log-likelihood is above 6.0 bits per
sub-pixel. At no point during training does the unregularized model get a test-set log-likelihood below 3.0 bits per sub-pixel. Contrary to what we might naively expect, the perceptual quality of the generated images by the overfitted model is not great, as shown in Figure 8.
4 CONCLUSION
We presented PixelCNN++, a modification of PixelCNN using a discretized logistic mixture likelihood on the pixels among other modifications. We demonstrated the usefulness of these modifications with state-of-the-art results on CIFAR-10. Our code is made available at https: //github.com/openai/pixel-cnn and can easily be adapted for use on other data sets.

Apologies for the late submission of this review, and thank you for the author’s responses to earlier questions.
This submission proposes an improved implementation of the PixelCNN generative model. Most of the improvements are small and can be considered as specific technical details such as the use of dropout and skip connections, while others are slightly more substantial such as the use of a different likelihood model and multiscale analysis. The submission demonstrates state-of-the-art likelihood results on CIFAR-10.
My summary of the main contribution:
Autoregressive-type models - of which PixelCNN is an example - are a nice class of models as their likelihood can be evaluated in closed form. A main differentiator for this type of models is how the conditional likelihood of one pixel conditioned on its causal neighbourhood is modelled:
- In one line of work such as (Theis et al, 2012 MCGSM, Theis et al 2015 Spatial LSTM) the conditional distribution is modelled as a continuous density over real numbers. This approach has limitations: We know that in observed data pixel intensities are quantized to a discrete integer representation so a discrete distribution could give better likelihoods. Furthermore these continuous distributions have a tail and assign some probability mass outside the valid range of pixel intensities, which may hurt the likelihood.
- In more recent work by van den Oord and colleagues the conditional likelihood is modelled as an arbitrary discrete distribution over the 256 possible values for pixel intensities. This does not suffer from the limitations of continuous likelihoods, but it also seems wasteful and is not very data efficient.
The authors propose something in the middle by keeping the discretized nature of the conditional likelihood, but restricting the discrete distribution to ones whose CDF that can be modeled as a linear combination of sigmoids. This approach makes sense to me, and is new in a way, but it doesn’t appear to be very revolutionary or significant to me.
The second somewhat significant modification is the use of downsampling and multiscale modelling (as opposed to dilated convolutions). The main motivation for the authors to do this is saving computation time while keeping the multiscale flexibility of the model. The authors also introduce shortcut connections to compensate for the potential loss of information as they perform downsampling. Again, I feel that this modification not particularly revolutionary. Multiscale image analysis with autoregressive generative models has been done for example in (Theis et al, 2012) and several other papers.
Overall I felt that this submission falls short on presenting substantially new ideas, and reads more like documentation for a particular implementation of an existing idea.


QUASI-RECURRENT NEURAL NETWORKS
Recurrent neural networks are a powerful tool for modeling sequential data, but the dependence of each timestep’s computation on the previous timestep’s output limits parallelism and makes RNNs unwieldy for very long sequences. We introduce quasi-recurrent neural networks (QRNNs), an approach to neural sequence modeling that alternates convolutional layers, which apply in parallel across timesteps, and a minimalist recurrent pooling function that applies in parallel across channels. Despite lacking trainable recurrent layers, stacked QRNNs have better predictive accuracy than stacked LSTMs of the same hidden size. Due to their increased parallelism, they are up to 16 times faster at train and test time. Experiments on language modeling, sentiment classification, and character-level neural machine translation demonstrate these advantages and underline the viability of QRNNs as a basic building block for a variety of sequence tasks.
1 INTRODUCTION
Recurrent neural networks (RNNs), including gated variants such as the long short-term memory (LSTM) (Hochreiter & Schmidhuber, 1997) have become the standard model architecture for deep learning approaches to sequence modeling tasks. RNNs repeatedly apply a function with trainable parameters to a hidden state. Recurrent layers can also be stacked, increasing network depth, representational power and often accuracy. RNN applications in the natural language domain range from sentence classification (Wang et al., 2015) to word- and character-level language modeling (Zaremba et al., 2014). RNNs are also commonly the basic building block for more complex models for tasks such as machine translation (Bahdanau et al., 2015; Luong et al., 2015; Bradbury & Socher, 2016) or question answering (Kumar et al., 2016; Xiong et al., 2016). Unfortunately standard RNNs, including LSTMs, are limited in their capability to handle tasks involving very long sequences, such as document classification or character-level machine translation, as the computation of features or states for different parts of the document cannot occur in parallel.
Convolutional neural networks (CNNs) (Krizhevsky et al., 2012), though more popular on tasks involving image data, have also been applied to sequence encoding tasks (Zhang et al., 2015). Such models apply time-invariant filter functions in parallel to windows along the input sequence. CNNs possess several advantages over recurrent models, including increased parallelism and better scaling to long sequences such as those often seen with character-level language data. Convolutional models for sequence processing have been more successful when combined with RNN layers in a hybrid architecture (Lee et al., 2016), because traditional max- and average-pooling approaches to combining convolutional features across timesteps assume time invariance and hence cannot make full use of large-scale sequence order information.
We present quasi-recurrent neural networks for neural sequence modeling. QRNNs address both drawbacks of standard models: like CNNs, QRNNs allow for parallel computation across both timestep and minibatch dimensions, enabling high throughput and good scaling to long sequences. Like RNNs, QRNNs allow the output to depend on the overall order of elements in the sequence. We describe QRNN variants tailored to several natural language tasks, including document-level sentiment classification, language modeling, and character-level machine translation. These models outperform strong LSTM baselines on all three tasks while dramatically reducing computation time.
∗Equal contribution
2 MODEL
Each layer of a quasi-recurrent neural network consists of two kinds of subcomponents, analogous to convolution and pooling layers in CNNs. The convolutional component, like convolutional layers in CNNs, allows fully parallel computation across both minibatches and spatial dimensions, in this case the sequence dimension. The pooling component, like pooling layers in CNNs, lacks trainable parameters and allows fully parallel computation across minibatch and feature dimensions.
Given an input sequence X ∈ RT×n of T n-dimensional vectors x1 . . .xT , the convolutional subcomponent of a QRNN performs convolutions in the timestep dimension with a bank of m filters, producing a sequence Z ∈ RT×m of m-dimensional candidate vectors zt. In order to be useful for tasks that include prediction of the next token, the filters must not allow the computation for any given timestep to access information from future timesteps. That is, with filters of width k, each zt depends only on xt−k+1 through xt. This concept, known as a masked convolution (van den Oord et al., 2016a), is implemented by padding the input to the left by the convolution’s filter size minus one.
We apply additional convolutions with separate filter banks to obtain sequences of vectors for the elementwise gates that are needed for the pooling function. While the candidate vectors are passed through a tanh nonlinearity, the gates use an elementwise sigmoid. If the pooling function requires a forget gate ft and an output gate ot at each timestep, the full set of computations in the convolutional component is then:
Z = tanh(Wz ∗X) F = σ(Wf ∗X) O = σ(Wo ∗X),
(1)
where Wz ,Wf , and Wo, each in Rk×n×m, are the convolutional filter banks and ∗ denotes a masked convolution along the timestep dimension. Note that if the filter width is 2, these equations reduce to the LSTM-like
zt = tanh(W 1 zxt−1 +W 2 zxt)
ft = σ(W 1 fxt−1 +W 2 fxt)
ot = σ(W 1 oxt−1 +W 2 oxt).
(2)
Convolution filters of larger width effectively compute higher n-gram features at each timestep; thus larger widths are especially important for character-level tasks.
Suitable functions for the pooling subcomponent can be constructed from the familiar elementwise gates of the traditional LSTM cell. We seek a function controlled by gates that can mix states across timesteps, but which acts independently on each channel of the state vector. The simplest option, which Balduzzi & Ghifary (2016) term “dynamic average pooling”, uses only a forget gate:
ht = ft ht−1 + (1− ft) zt, (3)
where denotes elementwise multiplication. The function may also include an output gate:
ct = ft ct−1 + (1− ft) zt ht = ot ct.
(4)
Or the recurrence relation may include an independent input and forget gate:
ct = ft ct−1 + it zt ht = ot ct.
(5)
We term these three options f -pooling, fo-pooling, and ifo-pooling respectively; in each case we initialize h or c to zero. Although the recurrent parts of these functions must be calculated for each timestep in sequence, their simplicity and parallelism along feature dimensions means that, in practice, evaluating them over even long sequences requires a negligible amount of computation time.
A single QRNN layer thus performs an input-dependent pooling, followed by a gated linear combination of convolutional features. As with convolutional neural networks, two or more QRNN layers should be stacked to create a model with the capacity to approximate more complex functions.
2.1 VARIANTS
Motivated by several common natural language tasks, and the long history of work on related architectures, we introduce several extensions to the stacked QRNN described above. Notably, many extensions to both recurrent and convolutional models can be applied directly to the QRNN as it combines elements of both model types.
Regularization An important extension to the stacked QRNN is a robust regularization scheme inspired by recent work in regularizing LSTMs.
The need for an effective regularization method for LSTMs, and dropout’s relative lack of efficacy when applied to recurrent connections, led to the development of recurrent dropout schemes, including variational inference–based dropout (Gal & Ghahramani, 2016) and zoneout (Krueger et al., 2016). These schemes extend dropout to the recurrent setting by taking advantage of the repeating structure of recurrent networks, providing more powerful and less destructive regularization.
Variational inference–based dropout locks the dropout mask used for the recurrent connections across timesteps, so a single RNN pass uses a single stochastic subset of the recurrent weights. Zoneout stochastically chooses a new subset of channels to “zone out” at each timestep; for these channels the network copies states from one timestep to the next without modification.
As QRNNs lack recurrent weights, the variational inference approach does not apply. Thus we extended zoneout to the QRNN architecture by modifying the pooling function to keep the previous pooling state for a stochastic subset of channels. Conveniently, this is equivalent to stochastically setting a subset of the QRNN’s f gate channels to 1, or applying dropout on 1− f :
F = 1− dropout(1− σ(Wf ∗X))) (6) Thus the pooling function itself need not be modified at all. We note that when using an off-theshelf dropout layer in this context, it is important to remove automatic rescaling functionality from the implementation if it is present. In many experiments, we also apply ordinary dropout between layers, including between word embeddings and the first QRNN layer.
Densely-Connected Layers We can also extend the QRNN architecture using techniques introduced for convolutional networks. For sequence classification tasks, we found it helpful to use skip-connections between every QRNN layer, a technique termed “dense convolution” by Huang et al. (2016). Where traditional feed-forward or convolutional networks have connections only between subsequent layers, a “DenseNet” with L layers has feed-forward or convolutional connections between every pair of layers, for a total ofL(L−1). This can improve gradient flow and convergence properties, especially in deeper networks, although it requires a parameter count that is quadratic in the number of layers.
When applying this technique to the QRNN, we include connections between the input embeddings and every QRNN layer and between every pair of QRNN layers. This is equivalent to concatenating
each QRNN layer’s input to its output along the channel dimension before feeding the state into the next layer. The output of the last layer alone is then used as the overall encoding result.
Encoder–Decoder Models To demonstrate the generality of QRNNs, we extend the model architecture to sequence-to-sequence tasks, such as machine translation, by using a QRNN as encoder and a modified QRNN, enhanced with attention, as decoder. The motivation for modifying the decoder is that simply feeding the last encoder hidden state (the output of the encoder’s pooling layer) into the decoder’s recurrent pooling layer, analogously to conventional recurrent encoder–decoder architectures, would not allow the encoder state to affect the gate or update values that are provided to the decoder’s pooling layer. This would substantially limit the representational power of the decoder.
Instead, the output of each decoder QRNN layer’s convolution functions is supplemented at every timestep with the final encoder hidden state. This is accomplished by adding the result of the convolution for layer ` (e.g., W`z ∗X`, in RT×m) with broadcasting to a linearly projected copy of layer `’s last encoder state (e.g., V`zh̃ ` T , in Rm):
Z` = tanh(W`z ∗X` +V`zh̃`T ) F` = σ(W`f ∗X` +V`f h̃`T )
O` = σ(W`o ∗X` +V`oh̃`T ),
(7)
where the tilde denotes that h̃ is an encoder variable. Encoder–decoder models which operate on long sequences are made significantly more powerful with the addition of soft attention (Bahdanau et al., 2015), which removes the need for the entire input representation to fit into a fixed-length encoding vector. In our experiments, we computed an attentional sum of the encoder’s last layer’s hidden states. We used the dot products of these encoder hidden states with the decoder’s last layer’s un-gated hidden states, applying a softmax along the encoder timesteps, to weight the encoder states into an attentional sum kt for each decoder timestep. This context, and the decoder state, are then fed into a linear layer followed by the output gate:
αst = softmax all s
(cLt · h̃Ls ) kt = ∑ s αsth̃ L s
hLt = ot (Wkkt +WccLt ),
(8)
where L is the last layer. This procedure is closely analogous to the attention mechanism described by Luong et al. (2015) as “global-dot without input feeding”. The reason for avoiding input feeding is to allow the QRNN layers to run in a maximally timestep-parallel way during training, even
Model Time / Epoch (s) Test Acc (%)
if they can’t during inference; any kind of input feeding would make this impossible, although it would likely result in slightly better translation performance.
While the first step of this attention procedure is quadratic in the sequence length, in practice it takes significantly less computation time than the model’s linear and convolutional layers due to the simple and highly parallel dot-product scoring function.
3 EXPERIMENTS
We evaluate the performance of the QRNN on three different natural language tasks: document-level sentiment classification, language modeling, and character-based neural machine translation. Our QRNN models outperform LSTM-based models of equal hidden size on all three tasks while dramatically improving computation speed. Experiments were implemented in Chainer (Tokui et al.).
3.1 SENTIMENT CLASSIFICATION
We evaluate the QRNN architecture on a popular document-level sentiment classification benchmark, the IMDb movie review dataset (Maas et al., 2011). The dataset consists of a balanced sample of 25,000 positive and 25,000 negative reviews, divided into equal-size train and test sets, with an average document length of 231 words (Wang & Manning, 2012). We compare only to other results that do not make use of additional unlabeled data (thus excluding e.g., Miyato et al. (2016)).
Our best performance on a held-out development set was achieved using a four-layer denselyconnected QRNN with 256 units per layer and word vectors initialized using 300-dimensional cased GloVe embeddings (Pennington et al., 2014). Dropout of 0.3 was applied between layers, and we used L2 regularization of 4 × 10−6. Optimization was performed on minibatches of 24 examples using RMSprop (Tieleman & Hinton, 2012) with learning rate of 0.001, α = 0.9, and = 10−8.
Small batch sizes and long sequence lengths provide an ideal situation for demonstrating the QRNN’s performance advantages over traditional recurrent architectures. This is because traditional architectures are only fully parallel over the batch dimension, while the QRNN parallelizes over batch and timestep dimensions in the convolutional layer and over batch and feature dimensions in the pooling layer. We observed a speedup of 3.2x on IMDb train time per epoch compared to the optimized LSTM implementation provided in NVIDIA’s cuDNN library. For specific batch sizes and sequence lengths, a 16x speed gain is possible. Figure 4 provides extensive speed comparisons.
In Figure 3, we visualize the hidden state vectors cLt of the final QRNN layer on part of an example from the IMDb dataset. Even without any post-processing, changes in the hidden state are visible and interpretable in regards to the input. This is a consequence of the elementwise nature of the recurrent pooling function, which delays direct interaction between different channels of the hidden state until the computation of the next QRNN layer.
3.2 LANGUAGE MODELING
We replicate the language modeling experiment of Zaremba et al. (2014) and Gal & Ghahramani (2016) to benchmark the QRNN architecture for natural language sequence prediction. The experiment uses a standard preprocessed version of the Penn Treebank (PTB) by Mikolov et al. (2010).
We implemented a gated QRNN model with medium hidden size: 2 layers with 640 units in each layer. Both QRNN layers use a convolutional filter width k of two timesteps. While the “medium” models used in other work (Zaremba et al., 2014; Gal & Ghahramani, 2016) consist of 650 units in each layer, it was more computationally convenient to use a multiple of 32. As the Penn Treebank is a relatively small dataset, preventing overfitting is of considerable importance and a major focus of recent research. It is not obvious in advance which of the many RNN regularization schemes would perform well when applied to the QRNN. Our tests showed encouraging results from zoneout applied to the QRNN’s recurrent pooling layer, implemented as described in Section 2.1.
The experimental settings largely followed the “medium” setup of Zaremba et al. (2014). Optimization was performed by stochastic gradient descent (SGD) without momentum. The learning rate was set at 1 for six epochs, then decayed by 0.95 for each subsequent epoch, for a total of 72 epochs. We additionally used L2 regularization of 2 × 10−4 and rescaled gradients with norm above 10. Zoneout was applied by performing dropout with ratio 0.1 on the forget gates of the QRNN, without rescaling the output of the dropout function. Batches consist of 20 examples, each 105 timesteps.
Model Parameters Validation Test
Comparing our results on the gated QRNN with zoneout to the results of LSTMs with both ordinary and variational dropout in Table 2, we see that the QRNN is highly competitive. The QRNN without zoneout strongly outperforms both our medium LSTM and the medium LSTM of Zaremba et al. (2014) which do not use recurrent dropout and is even competitive with variational LSTMs. This may be due to the limited computational capacity that the QRNN’s pooling layer has relative to the LSTM’s recurrent weights, providing structural regularization over the recurrence.
Without zoneout, early stopping based upon validation loss was required as the QRNN would begin overfitting. By applying a small amount of zoneout (p = 0.1), no early stopping is required and the QRNN achieves competitive levels of perplexity to the variational LSTM of Gal & Ghahramani (2016), which had variational inference based dropout of 0.2 applied recurrently. Their best performing variation also used Monte Carlo (MC) dropout averaging at test time of 1000 different masks, making it computationally more expensive to run.
When training on the PTB dataset with an NVIDIA K40 GPU, we found that the QRNN is substantially faster than a standard LSTM, even when comparing against the optimized cuDNN LSTM. In Figure 4 we provide a breakdown of the time taken for Chainer’s default LSTM, the cuDNN LSTM, and QRNN to perform a full forward and backward pass on a single batch during training of the RNN LM on PTB. For both LSTM implementations, running time was dominated by the RNN computations, even with the highly optimized cuDNN implementation. For the QRNN implementation, however, the “RNN” layers are no longer the bottleneck. Indeed, there are diminishing returns from further optimization of the QRNN itself as the softmax and optimization overhead take equal or greater time. Note that the softmax, over a vocabulary size of only 10,000 words, is relatively small; for tasks with larger vocabularies, the softmax would likely dominate computation time.
It is also important to note that the cuDNN library’s RNN primitives do not natively support any form of recurrent dropout. That is, running an LSTM that uses a state-of-the-art regularization scheme at cuDNN-like speeds would likely require an entirely custom kernel.
3.3 CHARACTER-LEVEL NEURAL MACHINE TRANSLATION
We evaluate the sequence-to-sequence QRNN architecture described in 2.1 on a challenging neural machine translation task, IWSLT German–English spoken-domain translation, applying fully character-level segmentation. This dataset consists of 209,772 sentence pairs of parallel training data from transcribed TED and TEDx presentations, with a mean sentence length of 103 characters for German and 93 for English. We remove training sentences with more than 300 characters in English or German, and use a unified vocabulary of 187 Unicode code points.
Our best performance on a development set (TED.tst2013) was achieved using a four-layer encoder– decoder QRNN with 320 units per layer, no dropout or L2 regularization, and gradient rescaling to a maximum magnitude of 5. Inputs were supplied to the encoder reversed, while the encoder convolutions were not masked. The first encoder layer used convolutional filter width k = 6, while
Model Train Time BLEU (TED.tst2014)
the other encoder layers used k = 2. Optimization was performed for 10 epochs on minibatches of 16 examples using Adam (Kingma & Ba, 2014) with α = 0.001, β1 = 0.9, β2 = 0.999, and = 10−8. Decoding was performed using beam search with beam width 8 and length normalization α = 0.6. The modified log-probability ranking criterion is provided in the appendix.
Results using this architecture were compared to an equal-sized four-layer encoder–decoder LSTM with attention, applying dropout of 0.2. We again optimized using Adam; other hyperparameters were equal to their values for the QRNN and the same beam search procedure was applied. Table 3 shows that the QRNN outperformed the character-level LSTM, almost matching the performance of a word-level attentional baseline.
4 RELATED WORK
Exploring alternatives to traditional RNNs for sequence tasks is a major area of current research. Quasi-recurrent neural networks are related to several such recently described models, especially the strongly-typed recurrent neural networks (T-RNN) introduced by Balduzzi & Ghifary (2016). While the motivation and constraints described in that work are different, Balduzzi & Ghifary (2016)’s concepts of “learnware” and “firmware” parallel our discussion of convolution-like and pooling-like subcomponents. As the use of a fully connected layer for recurrent connections violates the constraint of “strong typing”, all strongly-typed RNN architectures (including the T-RNN, T-GRU, and T-LSTM) are also quasi-recurrent. However, some QRNN models (including those with attention or skip-connections) are not “strongly typed”. In particular, a T-RNN differs from a QRNN as described in this paper with filter size 1 and f -pooling only in the absence of an activation function on z. Similarly, T-GRUs and T-LSTMs differ from QRNNs with filter size 2 and fo- or ifo-pooling respectively in that they lack tanh on z and use tanh rather than sigmoid on o.
Another related sequence model is the query-reduction network introduced by Seo et al. (2016). Such a network without its query component could be rewritten as a QRNN with filter size 1, while the full QRN is similar to a single layer of the decoder component of our sequence-to-sequence architecture.
The PixelCNN model (van den Oord et al., 2016a) was the first to tackle a sequence prediction problem (in particular, the computer vision equivalent of language modeling) using masked convolutions in place of recurrent units. Like the QRNN, the PixelCNN architecture allows for highly parallel computation whenever the whole input is available ahead of time (e.g., during training). In order to enable conditional image generation (a setting similar to the QRNN encoder–decoder), the outputs of the convolutions in a PixelCNN can be augmented with a term that depends on the encoder state, while better generation performance was obtained by adding an elementwise gate to the model output (van den Oord et al., 2016b). The PixelCNN, however, relies on depth and large filter sizes to provide long-term context dependence; unlike the QRNN, the gating mechanism is not recurrent.
1Unpublished result from NIPS 2016 tutorial by Nal Kalchbrenner, given after the submission of this paper (slides at https://drive.google.com/file/d/0B7jhGCaUwDJeZWZWUXJ4cktxVU0/view). See Related Work for discussion.
The QRNN encoder–decoder model shares the favorable parallelism and path-length properties exhibited by the ByteNet (Kalchbrenner et al., 2016), a PixelCNN-like architecture for character-level machine translation based on residual convolutions over binary trees. Their model was constructed to achieve three desired properties: parallelism, linear-time computational complexity, and short paths between any pair of words in order to better propagate gradient signals. While the ByteNet outperforms the QRNN encoder–decoder by about five BLEU points on the IWSLT dataset, it is unclear how much of this difference can be attributed to the overall ByteNet model architecture, as opposed to the many other contributions of that paper, like residual multiplicative blocks or subbatch normalization.
The QRNN is also related to work in hybrid convolutional–recurrent models. Zhou et al. (2015) apply CNNs at the word level to generate n-gram features used by an LSTM for text classification. Xiao & Cho (2016) also tackle text classification by applying convolutions at the character level, with a stride to reduce sequence length, then feeding these features into a bidirectional LSTM. A similar approach was taken by Lee et al. (2016) for character-level machine translation. Their model’s encoder uses a convolutional layer followed by max-pooling to reduce sequence length, a four-layer highway network, and a bidirectional GRU. The parallelism of the convolutional, pooling, and highway layers allows training speed comparable to subword-level models without hard-coded text segmentation.
5 CONCLUSION
Intuitively, many aspects of the semantics of long sequences are context-invariant and can be computed in parallel (e.g., convolutionally), but some aspects require long-distance context and must be computed recurrently. Many existing neural network architectures either fail to take advantage of the contextual information or fail to take advantage of the parallelism. QRNNs exploit both parallelism and context, exhibiting advantages from both convolutional and recurrent neural networks. QRNNs have better predictive accuracy than LSTM-based models of equal hidden size, even though they use fewer parameters and run substantially faster. Our experiments show that the speed and accuracy advantages remain consistent across tasks and at both word and character levels.
Extensions to both CNNs and RNNs are often directly applicable to the QRNN, while the model’s hidden states are more interpretable than those of other recurrent architectures as its channels maintain their independence across timesteps. We believe that QRNNs can serve as a building block for long-sequence tasks that were previously impractical with traditional RNNs.
APPENDIX
BEAM SEARCH RANKING CRITERION
The modified log-probability ranking criterion we used in beam search for translation experiments is:
log(Pcand) = T + α
T . . .
Ttrg + α
Ttrg T∑ i=1 log(p(wi|w1 . . . wi−1)), (9)
where α is a length normalization parameter (Wu et al., 2016), wi is the ith output character, and Ttrg is a “target length” equal to the source sentence length plus five characters. This reduces at α = 0 to ordinary beam search with probabilities:
log(Pcand) = T∑ i=1 log(p(wi|w1 . . . wi−1)), (10)
and at α = 1 to beam search with probabilities normalized by length (up to the target length):
log(Pcand) ∼ 1
T T∑ i=1 log(p(wi|w1 . . . wi−1)). (11)
Conveniently, this ranking criterion can be computed at intermediate beam-search timesteps, obviating the need to apply a separate reranking on complete hypotheses.
RESULTS ON COPY AND ADDITION TASKS
We take the addition task to mean sequence-to-sequence decimal addition of unaligned, variablelength numbers (the hardest of several versions). An example of this task for maximum length 5 digits is “73952+9462”→“83414”. We train on ntrain randomly generated examples for up to 100 epochs with early stopping, and report the smallest model setup that achieves> 99% character-level validation accuracy, or the best validation accuracy achieved by any model setup if none achieve 99%.
For ndigits = 5 and ntrain = 100000, the QRNN converges with models larger than 3 layers of 256 units, while in our experiments LSTMs require only 2 layers of 128 units. For ndigits = 10 and ntrain = 100000, an LSTM reaches 98.5% with 3 layers of 1024 units, while the best QRNN model (4 layers of 512 units) reaches only 95.0%.
The copy task is similarly implemented as sequence-to-sequence reconstruction of variable-length decimal numbers. For 5 digits, an example is “23532”→“23532”. We train on 10000 randomly generated examples for up to 100 epochs. For ndigits = 5, the QRNN converges with models larger than 2 layers of 32 units or 1 layer of 256 units, while the LSTM requires only 1 layer of 32 units. For ndigits = 10, the QRNN requires a model larger than 2 layers of 128 units while the LSTM requires a model of at least 2 layers of 64 units or 1 layer of 256 units. For ndigits = 40, a QRNN with 5 layers of 512 units reaches 98.0% while the best LSTM model, with 3 layers of 512 units, only reaches 95.5%.
A deeper LSTM would likely need some kind of residual or highway connections to converge on this task, while the deep QRNN converges relatively well despite our earlier experiences with 4-layer QRNNs without dense connections not converging successfully on the sentiment task.

This paper introduces a novel RNN architecture named QRNN.
QNNs are similar to gated RNN , however their gate and state update  functions depend only on the recent input values, it does not depend on the previous hidden state. The gate and state update functions are computed through a temporal convolution applied on the input.
Consequently, QRNN allows for more parallel computation since they have less  operations in their hidden-to-hidden transition depending on the previous hidden state compared to a GRU or LSTM. However, they possibly loose in expressiveness relatively to those models. For instance, it is not clear how such a model deals with long-term dependencies without having to stack up several QRNN layers.
Various extensions of QRNN, leveraging Zoneout, Densely-connected or seq2seq with attention, are also proposed.
Authors evaluate their approach on various tasks and datasets (sentiment classification, world-level language modelling and character level machine translation).
Overall the paper is an enjoyable read and the proposed approach is interesting,
Pros:
- Address an important problem
- Nice empirical evaluation showing the benefit of their approach
- Demonstrate up to 16x speed-up relatively to a LSTM
Cons:
- Somewhat incremental novelty compared to (Balduzizi et al., 2016)
Few specific questions:
- Is densely layer necessary to obtain good result on the IMDB task. How does a simple 2-layer QRNN compare with 2-layer LSTM?
- How does the i-fo-ifo pooling perform comparatively?
- How does QRNN deal with long-term time depency? Did you try on it on simple toy task such as the copy or the adding task?