Recurrent neural networks (RNNs) have been used extensively and with increasing success to model various types of sequential data. Much of this progress has been achieved through devising recurrent units and architectures with the flexibility to capture complex statistics in the data, such as long range dependency or localized attention phenomena. However, while many sequential data (such as video, speech or language) can have highly variable information flow, most recurrent models still consume input features at a constant rate and perform a constant number of computations per time step, which can be detrimental to both speed and model capacity. In this paper, we explore a modification to existing recurrent units which allows them to learn to vary the amount of computation they perform at each step, without prior knowledge of the sequence’s time structure. We show experimentally that not only do our models require fewer operations, they also lead to better performance overall on evaluation tasks.
1 INTRODUCTION
The class of Recurrent Neural Network models (RNNs) is particularly well suited to dealing with sequential data, and has been successfully applied to a diverse array of tasks, such as language modeling and speech recognition (Mikolov, 2012), machine translation (Mikolov, 2012; Cho et al., 2014a), or acoustic modeling (Robinson et al., 1993; Graves & Jaitly, 2014) among others. Two factors have been instrumental in allowing this paradigm to be so widely adopted and give rise to the aforementioned successes. On the one hand, recent advances in both hardware and software have had a significant role in bringing the training of recurrent models to tractable time periods. On the other hand, novel units and architectures have allowed recurrent networks to model certain features of sequential data better than Elman’s simple RNN architecture (Elman, 1990). These include such developments as the LSTM (Hochreiter & Schmidhuber, 1997) and GRU (Cho et al., 2014a) units, which can more easily learn to model long range interactions (Chung et al., 2014), or attention mechanisms that allow the model to focus on a specific part of its history when making a prediction (Bahdanau et al., 2014). In this work, we focus on another feature of recurrent networks: the ability to efficiently model processes happening at different and possibly varying time scales.
Most existing recurrent models take one of two approaches regarding the amount of computation they require. Either the computational load is constant over time, or it follows a fixed (or deterministic) schedule (Koutnı́k et al., 2014), (Mikolov et al., 2014). The latter approach has proven especially useful when dealing with sequences which reflect processes taking place at different lev-
∗Work done at Facebook AI Research
els (and time scales) (Bojanowski et al., 2015). However, we believe that taking a more flexible approach could prove useful.
Consider sequential data such as video feeds, audio signal, or language. In video data, there are time periods where the frames differ very slightly, and where the underlying model should probably do much less computation than when the scene completely changes. When modeling speech from an audio signal, it is also reasonable to expect that the model should be able do little to no computation during silences. Finally, in the case of character level language modeling, having more computational power at word boundaries can certainly help: after reading the left context The prime. . . , the model should be able to put a higher likelihood on the sequence of characters that make up the word minister. However, we can take this idea one step further: after reading The prime min. . . , the next few characters are almost deterministic, and the model should require little computation to predict the sequence i-s-t-e-r.
In this work, we show how to modify two commonly used recurrent unit architectures, namely the Elman and Gated Recurrent Unit, to obtain their variable computation counterparts. This gives rise to two new architecture, the Variable Computation RNN and Variable Computation GRU (VCRNN and VCGRU), which take advantage of these phenomena by deciding at each time step how much computation is required based on the current hidden state and input. We show that the models learn time patterns of interest, can perform fewer operations, and may even take advantage of these time structures to produce better predictions than the constant computation versions.
We start by giving an overview of related work in Section 2, provide background on the class of Recurrent Neural Networks in Section 3, describe our model and learning procedure in Section 4, and present experimental results on music as well as bit and character level language modeling in section 5. Finally, Section 6 concludes and lays out possible directions for future work.
2 RELATED WORK
How to properly handle sequences which reflect processes happening at different time scales has been a widely explored question. Among the proposed approaches, a variety of notable systems based on Hidden Markov Models (HMMs) have been put forward in the last two decades. The Factorial HMM model of (Ghahramani & Jordan, 1997) (and its infinite extension in (Gael et al., 2008)) use parallel interacting hidden states to model concurrent processes. While there is no explicit handling of different time scales, the model achieves good held-out likelihood on Bach chorales, which exhibit multi-scale behaviors. The hierarchical HMM model of (Fine et al., 1998) and (Murphy & Paskin, 2001) takes a more direct approach to representing multiple scales of processes. In these works, the higher level HMM can recursively call sub-HMMs to generate short sequences without changing its state, and the authors show a successful application to modeling cursive writing. Finally, the Switching State-Space Model of (Ghahramani & Hinton, 2000) combines HMMs and Linear Dynamical Systems: in this model, the HMM is used to switch between LDS parameters, and the experiments show that the HMM learns higher-level, slower dynamics than the LDS.
On the side of Recurrent Neural Networks, the idea that the models should have mechanisms that allow them to handle processes happening at different time scales is not a new one either. On the one hand, such early works as (Schmidhuber, 1991) and (Schmidhuber, 1992) already presented a two level architecture, with an “automatizer” acting on every time step and a “chunker” which should only be called when the automatizer fails to predict the next item, and which the author hypothesizes learns to model slower scale processes. On the other hand, the model proposed in (Mozer, 1993) has slow-moving units as well as regular ones, where the slowness is defined by a parameter τ ∈ [0, 1] deciding how fast the representation changes by taking a convex combination of the previous and predicted hidden state.
Both these notions, along with different approaches to multi-scale sequence modeling, have been developed in more recent work. (Mikolov et al., 2014) expand upon the idea of having slow moving units in an RNN by proposing an extension of the Elman unit which forces parts of the transition matrix to be close to the identity. The idea of having recurrent layers called at different time steps has also recently regained popularity. The Clockwork RNN of (Koutnı́k et al., 2014), for example, has RNN layers called every 1, 2, 4, 8, etc. . . time steps. The conditional RNN of (Bojanowski et al., 2015) takes another approach by using known temporal structure in the data: in the character level
level language modeling application, the first layer is called for every character, while the second is only called once per word. It should also be noted that state-of-the art results for language models have been obtained using multi-layer RNNs (Józefowicz et al., 2016), where the higher layers can in theory model slower processes. However, introspection in these models is more challenging, and it is difficult to determine whether they are actually exhibiting significant temporal behaviors.
Finally, even more recent efforts have considered using dynamic time schedules. (Chung et al., 2016) presents a multi-layer LSTM, where each layer decides whether or not to activate the next one at every time step. They show that the model is able to learn sensible time behaviors and achieve good perplexity on their chosen tasks. Another implementation of the general concept of adaptive timedependent computation is presented in (Graves, 2016). In that work, the amount of computation performed at each time step is varied not by calling units in several layers, but rather by having a unique RNN perform more than one update of the hidden state on a single time step. There too, the model can be shown to learn an intuitive time schedule.
In this paper, we present an alternative view of adaptive computation, where a single Variable Computation Unit (VCU) decides dynamically how much of its hidden state needs to change, leading to both savings in the number of operations per time step and the possibility for the higher dimensions of the hidden state to keep longer term memory.
3 RECURRENT NEURAL NETWORKS
Let us start by formally defining the class of Recurrent Neural Networks (RNNs). For tasks such as language modeling, we are interested in defining a probability distribution over sequences w = (w1, . . . , wT ). Using the chain rule, the negative log likelihood of a sequence can be written:
L(w) = − T∑ t=1 log ( p ( wt|F (w1, . . . , wt−1) )) . (1)
where F is a filtration, a function which summarizes all the relevant information from the past. RNNs are a class of models that can read sequences of arbitrary length to provide such a summary in the form of a hidden state ht ≈ F(w1, . . . , wt−1), by applying the same operation (recurrent unit) at each time step. More specifically, the recurrent unit is defined by a recurrence function g which takes as input the previous hidden state ht−1 at each time step t, as well as a representation of the input xt (where ht−1 and xt are D-dimensional vectors), and (with the convention h0 = 0,) outputs the new hidden state:
ht = g(ht−1, xt) (2)
Elman Unit. The unit described in (Elman, 1990) is often considered to be the standard unit. It is parametrized by U and V , which are square, D-dimensional transition matrices, and uses a sigmoid non-linearity to obtain the new hidden state:
ht = tanh(Uht−1 + V xt). (3)
In the Elman unit, the bulk of the computation comes from the matrix multiplications, and the cost per time step is O(D2). In the following section, we show a simple modification of the unit which allows it to reduce this cost significantly.
Gated Recurrent Unit. The Gated Recurrent Unit (GRU) was introduced in (Cho et al., 2014b). The main difference between the GRU and Elman unit consists in the model’s ability to interpolate between a proposed new hidden state and the current one, which makes it easier to model longer range dependencies. More specifically, at each time step t, the model computes a reset gate rt, an update gate zt, a proposed new hidden state h̃t and a final new hidden state ht as follows:
rt = σ(Urht−1 + Vrxt), zt = σ(Uzht−1 + Vzxt) (4)
h̃t = tanh(U(rt ht−1) + V xt) (5) And: ht = zt h̃t + (1− zt) ht−1 (6) Where denotes the element-wise product.
4 VARIABLE COMPUTATION RNN
As noted in the previous section, the bulk of the computation in the aforementioned settings comes from the linear layers; a natural option to reduce the number of operations would then be to only apply the linear transformations to a sub-set of the hidden dimensions. These could in theory correspond to any sub-set indices in {1, . . . , D}; however, we want a setting where the computational cost of the choice is much less than the cost of computing the new hidden state. Thus, we only consider the sets of first d dimensions of RD, so that there is a single parameter d to compute.
Our Variable Computation Units (VCUs) implement this idea using two modules: a scheduler decides how many dimensions need to be updated at the current time step, and the VCU performs a partial update of its hidden state accordingly, as illustrated in Figure 1. Section 4.1 formally describes the scheduler and partial update operations, and Section 4.2 outlines the procedure to jointly learn both modules.
4.1 MODEL DESCRIPTION
Scheduler. The model first needs to decide how much computation is required at the current time step. To make that decision, the recurrent unit has access to the current hidden state and input; this way, the model can learn to ignore an uninformative input, or to decide on more computation when an it is unexpected given the current hidden state. The scheduler is then defined as a function m : R2D → [0, 1] which decides what portion of the hidden state to change based on the current hidden and input vectors. In this work, we decide to implement it as a simple log-linear function with parameter vectors u and v, and bias b, and at each time step t, we have:
mt = σ(u · ht−1 + v · xt + b). (7)
Partial update. Once the scheduler has decided on a computation budget mt, the VCU needs to perform a partial update of the first dmtDe dimensions of its hidden state. Recall the hidden state ht−1 is a D-dimensional vector. Given a smaller dimension d ∈ {1, . . . , D}, a partial update of the hidden state would take the following form. Let gd be the d-dimensional version of the model’s recurrence function g as defined in Equation 2, which uses the upper left d by d square sub-matrices of the linear transformations (Ud, Vd, . . .), and hdt−1 and x d t denote the first d elements of ht−1 and xt. We apply gd to hdt−1 and x d t , and carry dimensions d + 1 to D from the previous hidden state, so the new hidden state ht is defined by:
hdt = gd(h d t−1, x d t ) and ∀i > d, ht,i = ht−1,i.
Soft mask. In practice, the transition function we just defined would require making a hard choice at each time step of the number of dimensions to be updated, which makes the model non-differentiable and can significantly complicate optimization. Instead, we approximate the hard choice by using a gate function to apply a soft mask. Given mt ∈ [0, 1] and a sharpness parameter λ, we use the gating vector et ∈ RD defined by:
∀i ∈ 1, . . . , D, (et)i = Thres ( σ ( λ(mtD − i) )) , (8)
where Thres maps all values greater than 1− and smaller than to 1 and 0 respectively. That way, the model performs an update using the first (mt ×D + η) dimensions of the hidden state, where η goes to 0 as λ increases, and leaves its last ((1−mt)×D − η) dimensions unchanged. Thus, if g is the recurrence function defined in Equation 2, we have:
h̄t−1 = et ht−1, īt = et xt, and ht = et g(h̄t−1, x̄t) + (1− et) ht. (9) The computational cost of this model at each step t, defined as the number of multiplications involving possibly non-zero elements is then O(m2tD
2). The construction of the VCRNN and VCGRU architectures using this method is described in the appendix.
4.2 LEARNING
Since the soft mask et is a continuous function of the model parameters, the scheduler can be learned through back-propagation. However, we have found that the naive approach of using a fixed sharpness parameter and simply minimizing the negative log-likelihood defined in Equation 1 led to the model being stuck in a local optimum which updates all dimensions at every step. We found that the following two modifications allowed the model to learn better parametrizations.
First, we can encourage mt to be either close or no greater than a target m̄ at all time by adding a penalty term Ω to the objective. For example, we can apply a `1 or `2 penalty to values of mt that are greater than the target, or that simply diverge from it (in which case we also discourage the model from using too few dimensions). The cost function defined in Equation 1 then becomes:
O(w, U, V,O, u, v, b) = L(w, U, V,O, u, v, b) + Ω(m, m̄). (10)
Secondly, for the model to be able to explore the effect of using fewer or more dimensions, we need to start training with a smooth mask (small λ parameter), since for small values of λ, the model actually uses the whole hidden state. We can then gradually increase the sharpness parameter until the model truly does a partial update.
5 EXPERIMENTS
We ran experiments with the Variable Computation variants of the Elman and Gated Recurrent Units (VCRNN and VCGRU respectively) on several sequence modeling tasks. All experiments were run using a symmetrical `1 penalty on the scheduler m, that is, penalizing mt when it is greater or smaller than target m̄, with m̄ taking various values in the range [0.2, 0.5]. In all experiments, we start with a sharpness parameter λ = 0.1, and increase it by 0.1 per epoch to a maximum value of 1.
In each of our experiments, we are interested in investigating two specific aspects of our model. On the one hand, do the time patterns that emerge agree with our intuition of the time dynamics expressed in the data? On the other hand, does the Variable Computation Unit (VCU) yield a good predictive model? More specifically, does it lead to lower perplexity than a constant computation counterpart which performs as many or more operations? In order to be able to properly assess the efficiency of the model, and since we do not know a priori how much computation the VCU uses, we always report the “equivalent RNN” dimension (noted as RNN-d in Table 3) along with the performance on test data, i.e. the dimension of an Elman RNN that would have performed the same amount of computation. Note that the computational complexity gains we refer to are exclusively in terms of lowering the number of operations, which does not necessarily correlate with a speed up of training when using general purpose GPU kernels; it is however a prerequisite to achieving such a speed up with the proper implementation, motivating our effort.
We answer both of these questions on the tasks of music modeling, bit and character level language modeling on the Penn Treebank text, and character level language modeling on the Text8 data set as well as two languages from the Europarl corpus.
5.1 MUSIC MODELING
We downloaded a corpus of Irish traditional tunes from https://thesession.org and split them into a training validation and test of 16,000 (2.4M tokens), 1,511 (227,000 tokens) and 2,000 (288,000 tokens) melodies respectively. Each sub-set includes variations of melodies, but no melody has variations across subsets. We consider each (pitch, length) pair to be a different symbol; with rests and bar symbols, this comes to a total vocabulary of 730 symbols.
Table 1 compares the perplexity on the test set to Elman RNNs with equivalent computational costs: an VCRNN with hidden dimension 500 achieves better perplexity with fewer operations than an RNN with dimension 250.
Looking at the output of the scheduler on the validation set also reveals some interesting patterns. First, bar symbols are mostly ignored: the average value of mt on bar symbols is 0.14, as opposed to 0.46 on all others. This is not surprising: our pre-processing does not handle polyphony or time signatures, so bars en up having different lengths. The best thing for the model to do is then just to ignore them and focus on the melody. Similarly, the model spends lest computation on rests (0.34 average mt), and pays less attention to repeated notes (0.51 average for mt on the first note of a repetition, 0.45 on the second).
We also notice that the model needs to do more computation on fast passages, which often have richer ornamentation, as illustrated in Table 2. While it is difficult to think a priori of all the sorts of behaviors that could be of interest, these initial results certainly show a sensible behavior of the scheduler on the music modeling task.
5.2 BIT AND CHARACTER LEVEL LANGUAGE MODELING
We also chose to apply our model to the tasks of bit level and character level language modeling. Those appeared as good applications since we know a priori what kind of temporal structure to look for: ASCII encoding means that we expect a significant change (change of character) every 8 bits in bit level modeling, and we believe the structure of word units to be useful when modeling text at the character level.
5.2.1 PENN TREEBANK AND TEXT8
We first ran experiments on two English language modeling tasks, using the Penn TreeBank and Text8 data sets. We chose the former as it is a well studied corpus, and one of the few corpora for which people have reported bit-level language modeling results. It is however quite small for our purposes, with under 6M characters, which motivated us to apply our models to the larger Text8 data set (100M characters). Table 3 shows bit per bit and bit per character results for bit and character level language modeling. We compare our results with those obtained with standard Elman RNN, GRU, and LSTM networks, as well as with the Conditional RNN of (Bojanowski et al., 2015).
Quantitative Results. We first compare the VCRNN to the regular Elman RNN, as well as to the Conditional RNN of (Bojanowski et al., 2015), which combines two layers running at bit and character level for bit level modeling, or character and word level for character level modeling. For bit level language modeling, the VCRNN not only performs fewer operations than the standard unit, it also achieves better performance. For character level modeling, the Elman model using a hidden dimension of 1024 achieved 1.47 bits per character, while our best performing VCRNN does slightly better while only requiring as much computation as a dimension 760 Elman unit. While we do slightly more computation than the Conditional RNN, it should be noted that our model is not explicitly given word-level information: it learns how to summarize it from character-level input.
The comparison between the constant computation and Variable Computation GRU (VCGRU) follows the same pattern, both on the PTB and Text8 corpora. On PTB, the VCGRU with the best validation perplexity performs as well as a GRU (and LSTM) of the same dimension with less than half the number of operations. On Text8, the VCGRU models with various values of the target m̄ always achieve better perplexity than other models performing similar or greater numbers of operations. It should be noted that none of the models we ran on Text8 overfits significantly (the training and validation perplexities are the same), which would indicate that the gain is not solely a matter of regularization.
Bit Level Scheduler. The scheduler in the bit level language model manages to learn the structure of ASCII encoding: Figure 2 shows that the higher dimensions are modified roughly every 8 bits. We also created some artificial data by taking the PTB text and adding 8 or 24 0 bits between each character. Figure 2, shows that the model learns to mostly ignore these “buffers”, doing most of its computation on actual characters.
Character Level Scheduler. On character level language modeling, the scheduler learns to make use of word boundaries and some language structures. Figure 3 shows that the higher dimensions are used about once per words, and in some cases, we even observe a spike at the end of each morpheme (long-stand-ing, as shown in Figure 5). While we provide results for the VCRNN specifically in this Section, the VCGRU scheduler follows the same patterns.
5.2.2 EUROPARL CZECH AND GERMAN
We also ran our model on two languages form the Europarl corpus. We chose Czech, which has a larger alphabet than other languages in the corpus, and German , which is a language that features long composite words without white spaces to indicate a new unit. Both are made up of about 20M characters. We tried two settings. In the “guide” setting, we use the penalty on mt to encourage the model to use more dimensions on white spaces. The “learn” setting is fully unsupervised, and encourages lower values of mt across the board.
Figure 4 shows that both perform similarly on the Czech dataset, achieving better performance more efficiently than the standard RNN. On German, the guided settings remains slightly more efficient than the fully learned one, but both are more efficient than the RNN and achieve the same performance when using more dimensions. Both learn to use more dimensions at word boundaries as shown in Figure 3. The German model also appears to be learning interesting morphology (Luftver-kehrs, eben-falls in Figure 3, An-satz, Um-welt-freund-lich in Figure 5), and grammar (focusing on case markers at the end of articles, Figure 5).
6 CONCLUSION AND FUTURE WORK
In this work, we have presented two kinds of Variable Computation recurrent units: the VCRNN and VCGRU, which modify the Elman and Gated Recurrent Unit respectively to allow the models to achieve better performance with fewer operations, and can be shown to find time patterns of interest in sequential data. We hope that these encouraging results will open up paths for further exploration of adaptive computation paradigms in neural networks in general, which could lead to more computation-efficient models, better able to deal with varying information flow or multi-scale processes. We also see a few immediate possibilities for extensions of this specific model. For example, the same idea of adaptive computation can similarly be applied to yet other commonly used recurrent units, such as LSTMs, or to work within the different layers of a stacked architecture, and we are working on adapting our implementation to those settings. We also hope to investigate the benefits of using stronger supervision signals to train the scheduler, such as the entropy of the prediction, to hopefully push our current results even further.
A APPENDIX
A.1 NEW UNITS: VCRNN AND VCGRU
We apply the method outlined in the previous paragraph to two commonly used architecture. Recall that, given a proportion of dimensions to use mt ∈ [0, 1] and a sharpness parameter λ, we the gating vector et ∈ RD is defined as:
∀i ∈ 1, . . . , D, (et)i = Thres ( σ ( λ(mtD − i) )) . (11)
The masked versions of the previous hidden and current input vectors respectively are then:
h̄t−1 = et ht−1 and īt = et xt. (12)
First, we derive a variable computation version of the Elman RNN to get the Variable Computation Recurrent Neural Network (VCRNN) by transforming Equation 3 as follows:
ht = et g(h̄t−1, x̄t) + (1− et) ht. (13)
Secondly, we obtain the Variable Computation Gated Recurrent Unit (VCGRU) by deriving the variable computation of the GRU architecture. This is achieved by modifying Equations 4 to 6 as follows:
rt = σ(Urh̄t−1 + Vrx̄t), zt = et σ(Uzh̄t−1 + Vzx̄t) (14)
h̃t = tanh(U(rt h̄t−1) + V x̄t) (15) And:
ht = zt h̃t + (1− zt) ht−1 (16)

This paper describes a simple but clever method for allowing variable amounts of computation at each time step in RNNs. The new architecture seems to outperform vanilla RNNs on various sequence modelling tasks. Visualizations of the assignment of computational resources over time support the hypothesis that the model is able to learn to assign more computations whenever longer longer term dependencies need to be taken into account.
The proposed model is evaluated on a multitude of tasks and its ability to outperform similar architectures seems consistent. Some of the tasks allow for an interesting analysis of the amount of computation the model requests at each time step. It’s very interesting to see how the model seems to use more resources at the start of each word or ASCII character. I also like the investigation of the effect of imposing a pattern of computational budget assignment which uses prior knowledge about the task. The superior performance of the architecture is impressive but I’m not yet convinced that the baseline models had an equal number of hyperparameters to tune. I’ll come back to this point in the next paragraph because it’s mainly a clarity issue.
The abstract claims that the model is computationally more efficient than regular RNNs. There are no wall time measurements supporting this claim. While the model is theoretically able to save computations, the points made by the paper are clearly more conceptual and about the ability of the model to choose how to allocate its resources. This makes the paper interesting enough by itself but the claims of computational gains are misleading without actual results to back them up. I also find it unfortunate that it’s not clear from the text how the hyperparameter \bar{m} was chosen. Whether it was chosen randomly or set using a hyperparameter search on held-out data influences the fairness of a comparison with RNNs which did not have a similar type of hyperparameter for controlling regularization like for example dropout or weight noise (even if regularization of RNNs is a bit tricky). I don’t consider this a very serious flaw because I’m impressed enough by the fact that the new architecture achieves roughly similar performance while learning to allocate resources but I do think that details of this type are too important to be absent from the text. Even if the superior performance is due to this extra regularization controlling parameter it can actually be seen as a useful part of the architecture but it would be nice to know how sensitive the model is to its precise value.
To my knowledge, the proposed architecture is novel. The way the amount of computation is determined is unlike other methods for variable computation I have seen and quite inventive. Originality is one of this paper’s strongest points.
It’s currently hard to predict whether this method for variable computation will be used a lot in practice given that this also depends on how feasible it is to obtain actual computational gains at the hardware level. That said, the architecture may turn out to be useful for learning long-term dependencies. I also think that the interpretability of the value m_t is a nice property of the method and that it’s visualizations are very interesting. It might shed some more light into what makes certain tasks difficult for RNNs.
Pros:
Original clever idea.
Nice interesting visualizations.
Interesting experiments.
Cons:
Some experimental details are not clear.
I’m not convinced of the strength of the baseline.
The paper shouldn’t claim actual computational savings without reporting wall-clock times.
Edit:
I'm very positively impressed by the way the authors ended up addressing the biggest concerns I had about the paper and raised my score. Adding an LSTM baseline and results with a GRU version of the model significantly improves the empirical quality of the paper. On top of that, the authors addressed my question about some experimental detail I found important and promised to change the wording of the paper to remove confusion about whether the computational savings are conceptual or in actual wall time. I think it's fine that they are conceptual only as long as this is clear from the paper and abstract. I want to make clear to the AC that since the changes to the paper are currently still promises, my new score should be assumed to apply to an updated version of the paper in which the aforementioned concerns have indeed been addressed.
Edit:
Since I didn't know that the difference with the SOTA for some of these tasks was so large, I had to lower my score again after learning about this. I still think it's a good paper but with these results I cannot say that it stands out.


LEARNING IDENTITY MAPPINGS WITH RESIDUAL GATES
We propose a layer augmentation technique that adds shortcut connections with a linear gating mechanism, and can be applied to almost any network model. By using a scalar parameter to control each gate, we provide a way to learn identity mappings by optimizing only one parameter. We build upon the motivation behind Highway Neural Networks and Residual Networks, where a layer is reformulated in order to make learning identity mappings less problematic to the optimizer. The augmentation introduces only one extra parameter per layer, and provides easier optimization by making degeneration into identity mappings simpler. Experimental results show that augmenting layers provides better optimization, increased performance, and more layer independence. We evaluate our method on MNIST using fully-connected networks, showing empirical indications that our augmentation facilitates the optimization of deep models, and that it provides high tolerance to full layer removal: the model retains over 90% of its performance even after half of its layers have been randomly removed. In our experiments, augmented plain networks – which can be interpreted as simplified Highway Neural Networks – perform similarly to ResNets, raising new questions on how shortcut connections should be designed. We also evaluate our model on CIFAR-10 and CIFAR-100 using augmented Wide ResNets, achieving 3.65% and 18.27% test error, respectively.
We propose a layer augmentation technique that adds shortcut connections with a linear gating mechanism, and can be applied to almost any network model. By using a scalar parameter to control each gate, we provide a way to learn identity mappings by optimizing only one parameter. We build upon the motivation behind Highway Neural Networks and Residual Networks, where a layer is reformulated in order to make learning identity mappings less problematic to the optimizer. The augmentation introduces only one extra parameter per layer, and provides easier optimization by making degeneration into identity mappings simpler. Experimental results show that augmenting layers provides better optimization, increased performance, and more layer independence. We evaluate our method on MNIST using fully-connected networks, showing empirical indications that our augmentation facilitates the optimization of deep models, and that it provides high tolerance to full layer removal: the model retains over 90% of its performance even after half of its layers have been randomly removed. In our experiments, augmented plain networks – which can be interpreted as simplified Highway Neural Networks – perform similarly to ResNets, raising new questions on how shortcut connections should be designed. We also evaluate our model on CIFAR-10 and CIFAR-100 using augmented Wide ResNets, achieving 3.65% and 18.27% test error, respectively.
1 INTRODUCTION
As the number of layers of neural networks increase, effectively training its parameters becomes a fundamental problem (Larochelle et al. (2009)). Many obstacles challenge the training of neural networks, including vanishing/exploding gradients (Bengio et al. (1994)), saturating activation functions (Xu et al. (2016)) and poor weight initialization (Glorot & Bengio (2010)). Techniques such as unsupervised pre-training (Bengio et al. (2007)), non-saturating activation functions (Nair & Hinton (2010)) and normalization (Ioffe & Szegedy (2015)) target these issues and enable the training of deeper networks. However, stacking more than a dozen layers still lead to a hard to train model.
Recently, models such as Residual Networks (He et al. (2015b)) and Highway Neural Networks (Srivastava et al. (2015)) permitted the design of networks with hundreds of layers. A key idea of these models is to allow for information to flow more freely through the layers, by using shortcut connections between the layer’s input and output. This layer design greatly facilitates training, due to shorter paths between the lower layers and the network’s error function. In particular, these models can more easily learn identity mappings in the layers, thus allowing the network to be deeper
and learn more abstract representations (Bengio et al. (2012)). Such networks have been highly successful in many computer vision tasks.
On the theoretical side, it is suggested that depth contributes exponentially more to the representational capacity of networks than width (Eldan & Shamir (2015) Telgarsky (2016) Bianchini & Scarselli (2014) Montúfar et al. (2014)). This agrees with the increasing depth of winning architectures on challenges such as ImageNet (He et al. (2015b) Szegedy et al. (2014)).
Increasing the depth of networks significantly increases its representational capacity and consequently its performance, an observation supported by theory (Eldan & Shamir (2015) Telgarsky (2016) Bianchini & Scarselli (2014) Montúfar et al. (2014)) and practice (He et al. (2015b) Szegedy et al. (2014)). Moreover, He et al. (2015b) showed that, by construction, one can increase a network’s depth while preserving its performance. These two observations suggest that it suffices to stack more layers to a network in order to increase its performance. However, this behavior is not observed in practice even with recently proposed models, in part due to the challenge of training ever deeper networks.
In this work we aim to improve the training of deep networks by proposing a layer augmentation that builds on the idea of using shortcut connections, such as in Residual Networks and Highway Neural Networks. The key idea is to facilitate the learning of identity mappings by introducing a shortcut connection with a linear gating mechanism, as illustrated in Figure 1. Note that the shortcut connection is controlled by a gate that is parameterized with a scalar, k. This is a key difference from Highway Networks, where a tensor is used to regulate the shortcut connection, along with the incoming data. The idea of using a scalar is simple: it is easier to learn k = 0 than to learn Wg = 0 for a weight tensor Wg controlling the gate. Indeed, this single scalar allows for stronger supervision on lower layers, by making gradients flow more smoothly in the optimization.
We apply our proposed layer re-design to plain and residual layers, with the latter illustrated in Figure 2. Note that when augmenting a residual layer it becomes simply u = g(k)fr(x,W ) + x, where fr denotes the layer’s residual function. Thus, the shortcut connection allows the input to flow freely without any interference of g(k) through the layer. In the next sections we will call augmented plain networks (illustrated in Figure 1) Gated Plain Network and augmented residual networks (illustrated in Figure 2) Gated Residual Network, or GResNet. Again, note that in both cases learning identity mappings is much easier in comparison to the original models.
Note that layers that degenerated into identity mappings have no impact in the signal propagating through the network, and thus can be removed without affecting performance. The removal of such layers can be seen as a transposed application of sparse encoding (Glorot et al. (2011)): transposing the sparsity from neurons to layers provides a form to prune them entirely from the network. In-
deed, we show that performance decays slowly in GResNets when layers are removed, even when compared to ResNets.
We evaluate the performance of the proposed design in two experiments. First, we evaluate fullyconnected Gated PlainNets and Gated ResNets on MNIST and compare them with their nonaugmented counterparts, showing superior performance and robustness to layer removal. Second, we apply our layer re-design to Wide ResNets (Zagoruyko & Komodakis (2016)) and test its performance on CIFAR, obtaining results that are superior to all previously published results (to the best of our knowledge). These findings indicate that learning identity mappings is a fundamental aspect of learning in deep networks, and designing models where this is easier seems highly effective.
2 AUGMENTATION WITH RESIDUAL GATES
2.1 THEORETICAL INTUITION
Recall that a network’s depth can always be increased without affecting its performance – it suffices to add layers that perform identity mappings. Consider a plain fully-connected ReLU network with layers defined as u = ReLU(〈x,W 〉). When adding a new layer, if we initialize W to the identity matrix I , we have: u = ReLU(〈x, I〉) = ReLU(x) = x The last step holds since x is an output of a previous ReLU layer, and ReLU(ReLU(x)) = ReLU(x). Thus, adding more layers should only improve performance. However, how can a network with more layers learn to yield performance superior than a network with less layers? A key observation is that if learning identity mapping is easy, then the network with more layers is more likely to yield superior performance, as it can more easily recover the performance of a smaller network through identity mappings.
The layer design of Highway Neural Networks and Residual Networks allows for deeper models to be trained due to their shortcut connections. Note that in ResNets the identity mapping is learned
when W = 0 instead of W = I . Similarly, a Highway layer can degenerate into an identity mapping when the gating term T (x,WT ) equals zero for all data points. Since learning identity mappings in Highway Neural Networks strongly depends on the choice of the trasnform function T (and is non-trivial when T is the sigmoid function, since T−1(0) is not defined) we will focus our analysis on ResNets due to their simplicity. Considering a residual layer u = ReLU(〈x,W 〉) + x, we have:
u = ReLU(〈x, 0〉) + x = ReLU(0) + x = x
Intuitively, residual layers can degenerate into identity mappings more effectively since learning an all-zero matrix is easier than learning the identity matrix. To support this argument, consider weight parameters randomly initialized with zero mean. Hence, the point W = 0 is located exactly in the center of the probability mass distribution used to initialize the weights.
Recent work (Zhang et al. (2016)) suggests that the L2 norm of a critical point is an important factor regarding how easily the optimizer will reach it. More specifically, residual layers can be interpreted as a translation of the parameter set W = I to W = 0, which is more accessible in the optimization process due to its inferior L2 norm.
However, assuming that residual layers can trivially learn the parameter set W = 0 implies ignoring the randomness when initializing the weights. We demonstrate this by calculating the expected component-wise distance between Wo and the origin. Here, Wo denotes the weight tensor after initialization and prior to any optimization. Note that the distance between Wo and the origin captures the effort for a network to learn identity mappings:
E [ (Wo − 0)2 ] = E [ W 2o ] = V ar [ Wo ] Note that the distance is given by the distribution’s variance, and there is no reason to assume it to be negligible. Additionally, the fact that Residual Networks still suffer from optimization issues caused by depth (Huang et al. (2016a)) further supports this claim.
Some initialization schemes propose a variance in the order of O( 1n ) (Glorot & Bengio (2010), He et al. (2015a)), however this represents the distance for each individual parameter in W . For tensors with O(n2) parameters, the total distance – either absolute or Euclidean – between Wo and the origin will be in the order of O(n).
2.2 RESIDUAL GATES
As previously mentioned, the key contribution in this work is the proposal of a layer augmentation technique where learning a single scalar parameter suffices in order for the layer to degenerate into an identity mapping, thus making optimization easier for increased depths. As in Highway Networks, we propose the addition of gated shortcut connections. Our gates, however, are parameterized by a single scalar value, being easier to analyze and learn. For layers augmented with our technique, the effort required to learn identity mappings does not depend on any parameter, such as the layer width, in sharp contrast to prior models.
Our design is as follows: a layer u = f(x,W ) becomes u = g(k)f(x,W )+(1−g(k))x, where k is a scalar parameter. This design is illustrated in Figure 1. Note that such layer can quickly degenerate by setting g(k) to 0. Using the ReLU activation function as g, it suffices that k ≤ 0 for g(k) = 0. By adding an extra parameter, the dimensionality of the cost surface also grows by one. This new dimension, however, can be easily understood due to the specific nature of the layer reformulation. The original surface is maintained on the k = 1 slice, since the gated model becomes equivalent to the original one. On the k = 0 slice we have an identity mapping, and the associated cost for all points in such slice is the same cost associated with the point {k = 1,W = I}: this follows since both parameter configurations correspond to identity mappings, therefore being equivalent. Lastly, due to the linear nature of g(k) and consequently of the gates, all other slices k 6= 0, k 6= 1 will be a linear combination between the slices k = 0 and k = 1.
In addition to augmenting plain layers, we also apply our technique to residual layers. Although it might sound counterintuitive to add residual gates to a residual layer, we can see in Figure 2 that our augmentation provides ResNets means to regulate the residuals, therefore a linear gating
mechanism might not only allow deeper models, but could also improve performance. Having the original design of a residual layer as:
u = f(x,W ) = fr(x,W ) + x
where fr(x,W ) is the layer’s residual function – in our case, BN-ReLU-Conv-BN-ReLU-Conv. Our approach changes this layer by adding a linear gate, yielding:
u = g(k)f(x,W ) + (1− g(k))x = g(k)(fr(x,W ) + x) + (1− g(k))x = g(k)fr(x,W ) + x
The resulting layer maintains the shortcut connection unaltered, which according to He et al. (2016) is a desired property when designing residual blocks. As (1− g(k)) vanishes from the formulation, g(k) stops acting as a dual gating mechanism and can be interpreted as a flow regulator. Note that this model introduces a single scalar parameter per layer block. This new dimension can be interpreted as discussed above, except that the slice k = 0 is equivalent to {k = 1,W = 0}, since an identity mapping is learned when W = 0 in ResNets.
3 EXPERIMENTS
All models were implemented on Keras (Chollet (2015)) or on Torch (Collobert et al. (2011)), and were executed on a Geforce GTX 1070. Larger models or more complex datasets, such as the ImageNet (Russakovsky et al. (2015)), were not explored due to hardware limitations.
3.1 MNIST
The MNIST dataset (Lecun et al. (1998)) is composed of 60, 000 greyscale images with 28 × 28 pixels. Images represent handwritten digits, resulting in a total of 10 classes. We trained four types of fully-connected models: classical plain networks, ResNets, Gated Plain networks and Gated ResNets.
The networks consist of a linear layer with 50 neurons, followed by d layers with 50 neurons each, and lastly a softmax layer for classification. Only the d middle layers differ between the four architectures – the first linear layer and the softmax layer are the same in all experiments.
For plain networks, each layer performs dot product, followed by Batch Normalization and a ReLU activation function.
Initial tests with pre-activations (He et al. (2016)) resulted in poor performance on the validation set, therefore we opted for the traditional Dot-BN-ReLU layer when designing Residual Networks. Each residual block consists of two layers, as conventional.
All networks were trained using Adam (Kingma & Ba (2014)) with Nesterov momentum (Dozat) for a total of 100 epochs using mini-batches of size 128. No learning rate decay was used: we kept the learning rate and momentum fixed to 0.002 and 0.9 during the whole training.
For preprocessing, we divided each pixel value by 255, normalizing their values to [0, 1].
The training curves for plain networks, Gated PlainNets, ResNets and Gated ResNets with varying depth are shown in Figure 4. The distance between the curves increase with the depth, showing that the augmentation helps the training of deeper models.
Table 1 shows the test error for each depth and architecture. Augmented models perform better in all settings when compared to the original ones, and the performance boost is more noticeable with increased depths. Interestingly, Gated PlainNets performed better than ResNets, suggesting that the reason for Highway Neural Networks to underperform ResNets might be due to an overly complex gating mechanism.
As observed in Table 2, the mean values of k decrease as the model gets deeper, showing that shortcut connections have less impact on shallow networks. This agrees with empirical results that ResNets perform better than classical plain networks as the depth increases. Note that the significant difference between mean values for k in Gated PlainNets and Gated ResNets has an intuitive explanation: in order to suppress the residual signal against the shortcut connection, Gated PlainNets require that k < 0.5 (otherwise the residual signal will be enhanced). Conversely, Gated ResNets suppress the residual signal when k < 1.0, and enhance it otherwise.
We also analyzed how layer removal affects ResNets and Gated ResNets. We compared how the deepest networks (d = 100) behave as residual blocks composed of 2 layers are completely removed from the models. The final values for each k parameter, according to its corresponding residual block, is shown in Figure 5. We can observe that layers close to the middle of the network have a smaller k than these in the beginning or the end. Therefore, the middle layers have less importance by due to being closer to identity mappings.
Results are shown in Figure 5. For Gated Residual Networks, we prune pairs of layers following two strategies. One consists of pruning layers in a greedy fashion, where blocks with the smallest k are removed first. In the other we remove blocks randomly. We present results using both strategies for Gated ResNets, and only random pruning for ResNets since they lack the k parameter.
The greedy strategy is slightly better for Gated Residual Networks, showing that the k parameter is indeed a good indicator of a layer’s importance for the model, but that layers tend to assume the same level of significance. In a fair comparison, where both models are pruned randomly, Gated ResNets retain a satisfactory performance even after half of its layers have been removed, while ResNets suffer performance decrease after just a few layers.
Therefore augmented models are not only more robust to layer removal, but can have a fair share of their layers pruned and still perform well. Faster predictions can be generated by using a pruned version of an original model.
3.2 CIFAR
The CIFAR datasets (Krizhevsky (2009)) consists of 60, 000 color images with 32× 32 pixels each. CIFAR-10 has a total of 10 classes, including pictures of cats, birds and airplanes. The CIFAR-100 dataset is composed of the same number of images, however with a total of 100 classes.
Residual Networks have surpassed state-of-the-art results on CIFAR. We test Gated ResNets, Wide Gated ResNets (Zagoruyko & Komodakis (2016)) and compare them with their original, nonaugmented models.
For pre-activation ResNets, as described in He et al. (2016), we follow the original implementation details. We set an initial learning rate of 0.1, and decrease it by a factor of 10 after 50% and 75% epochs. SGD with Nesterov momentum of 0.9 are used for optimization, and the only preprocessing consists of mean subtraction. Weight decay of 0.0001 is used for regularization, and Batch Normalization’s momentum is set to 0.9.
We follow the implementation from Zagoruyko & Komodakis (2016) for Wide ResNets. The learning rate is initialized as 0.1, and decreases by a factor of 5 after 30%, 60% and 80% epochs. Images are mean/std normalized, and a weight decay of 0.0005 is used for regularization. We also apply 0.3 dropout (Srivastava et al. (2014)) between convolutions, whenever specified. All other details are the same as for ResNets.
For both architectures we use moderate data augmentation: images are padded with 4 pixels, and we take random crops of size 32× 32 during training. Additionally, each image is horizontally flipped with 50% probability. We use batch size 128 for all experiments.
For all gated networks, we initialize k with a constant value of 1. One crucial question is whether weight decay should be applied to the k parameters. We call this ”k decay”, and also compare Gated ResNets and Wide Gated ResNets when it is applied with the same magnitude of the weight decay: 0.0001 for Gated ResNet and 0.0005 for Wide Gated ResNet.
Table 3 shows the test error for two architectures: a ResNet with n = 5, and a Wide ResNet with n = 4, n = 10. Augmenting each model adds 15 and 12 parameters, respectively. We observe that k decay hurts performance in both cases, indicating that they should either remain unregularized or suffer a more subtle regularization compared to the weight parameters. Due to its direct connection to layer degeneration, regularizing k results in enforcing identity mappings, which might harm the model.
Due to the indications that a regularization on the k parameter results in a negative impact on the model’s performance, we proceed to test other models – having different depths and widening factors – with the goal of evaluating the effectiveness of our proposed augmentation. Tables 4 and 5 show that augmented Wide ResNets outperform the original models without changing any hyperparameter, both on CIFAR-10 and CIFAR-100.
As in the previous experiment, in Figure 6 we present the final k values for each block, in this case of the Wide ResNet (4,10) on CIFAR-10. We can observe that the k values follow an intriguing pattern: the lowest values are for the blocks of index 1, 5 and 9, which are exactly the ones that increase the feature map dimension. This indicates that, in such residual blocks, the convolution performed in the shortcut connection to increase dimension is more important than the residual block itself. Additionally, the peak value for the last residual block suggests that its shortcut connection is of little importance, and could as well be fully removed without greatly impacting the model.
Figure 7 shows the loss curves for Gated Wide ResNet (4,10) + Dropout, both on CIFAR-10 and CIFAR-100. The optimization behaves similarly to the original model, suggesting that the gates do
not have any side effects on the network. The performance gains presented on Table 4 point that, however predictable and extremely simple, our augmentation technique is powerful enough to aid on the optimization of state-of-the-art models.
Results of different models on the CIFAR datasets are shown in Table 6. The training and test errors are presented in Figure 7. To the authors’ knowledge, those are the best results on CIFAR-10 and CIFAR-100 with moderate data augmentation – only random flips and translations.
3.3 INTERPRETATION
Greff et al. (2016) showed how Residual and Highway layers can be interpreted as performing iterative refinements on learned representations. In this view, there is a connection on a layer’s learned parameters and the level of refinement applied on its input: for Highway Neural Networks, T (x) having components close to 1 results in a layer that generates completely new representations. As seen before, components close to 0 result in an identity mapping, meaning that the representations are not refined at all.
However, the dependency of T (x) on the incoming data makes it difficult to analyze the level of refinement performed by a layer given its parameters. This is more clearly observed once we consider how each component of T (x) is a function not only on the parameter set WT , but also on x.
In particular, given the mapping performed by a layer, we can estimate how much more abstract its representations are compared to the inputs. For our technique, this estimation can be done by observing the k parameter of the corresponding layer: in Gated PlainNets, k = 0 corresponds to an identity mapping, and therefore there is no modification on the learned representations. For k = 1, the shortcut connection is ignored and therefore a jump in the representation’s complexity is observed.
For Gated ResNets, the shortcut connection is never completely ignored in the generation of output. However, we can see that as k grows to infinity the shortcut connection’s contribution goes to zero, and the learned representation becomes more abstract compared to the layer’s inputs.
Table 6 shows how the layers that change the data dimensionality learn more abstract representations compared to dimensionality-preserving layers, which agrees with Greff et al. (2016). The last layer’s k value, which is the biggest among the whole model, indicates a severe jump in the abstraction of its representation, and is intuitive once we see the model as being composed of two main stages: a convolutional one and a fully-connected one, specific for classification.
Finally, Table 2 shows that the abstraction jumps decrease as the model grows deeper and the performance increases. This agrees with the idea that depth allows for more refined representations to be learned. We believe that an extensive analysis on the rate that these measures – depth, abstraction jumps and performance – interact with each other could bring further understanding on the practical benefits of depth in networks.
4 CONCLUSION
We have proposed a novel layer augmentation technique that facilitates the optimization of deep networks by making identity mappings easy to learn. Unlike previous models, layers augmented by our technique require optimizing only one parameter to degenerate into identity, and by designing our method such that randomly initialized parameter sets are always close to identity mappings, our design offers less optimization issues caused by depth.
Our experiments showed that augmenting plain and residual layers improves performance and facilitates learning in settings with increased depth. In the MNIST dataset, augmented plain networks outperformed ResNets, suggesting that models with gated shortcut connections – such as Highway Neural Networks – could be further improved by redesigning the gates.
We have shown that applying our technique to ResNets yield a model that can regulate the residuals. This model performed better in all our experiments with negligible extra training time and
parameters. Lastly, we have shown how it can be used for layer pruning, effectively removing large numbers of parameters from a network without necessarily harming its performance.

This paper proposes to learn a single scalar gating parameter instead of a full gating tensor in highway networks. The claim is that such gating is easier to learn and allows a network to flexibly utilize computation.
The basic idea of the paper is simple and is clearly presented. It is a natural simplification of highway networks to allow easily "shutting off" layers while keeping number of additional parameters low. However, in this regard the paper leaves out a few key points. Firstly, it does not mention that the gates in highway networks are data-dependent which is potentially more powerful than learning a fixed gate for all units and independent of data. Secondly, it does not do a fair comparison with highway networks to show that this simpler formulation is indeed easier to learn.
Did the authors try their original design of u = g(k)f(x) + (1 - g(k))x where f(x) is a plain layer instead of a residual layer? Based on the arguments made in the paper, this should work fine. Why wasn't it tested? If it doesn't work, are the arguments incorrect or incomplete?
For the MNIST experiments, since the hyperparameters are fixed, the plots are misleading if any dependence on hyperparameters exists for the different models. This experiment appears to be based on Srivastava et al (2015). If it is indeed designed to test optimization at aggressive depths, then apart from doing a hyperparameter search, the authors should not use regularization such as dropout or batch norm, which do not appear in the theoretical arguments for the architecture.
For CIFAR experiments, the obtained improvements compared to the baseline (wide resnets) are very small and therefore it is important to report the standard deviations (or all results) in both cases. It's not clear that the differences are significant.
Some questions regarding g(): Was g() always ReLU? Doesn't this have potential problems with g(k) becoming 0 and never recovering? Does this also mean that for the wide resnet in Fig 7, most residual blocks are zeroed out since k < 0?