INCREMENTAL NETWORK QUANTIZATION: TOWARDS LOSSLESS CNNS WITH LOW-PRECISION WEIGHTS
This paper presents incremental network quantization (INQ), a novel method, targeting to efficiently convert any pre-trained full-precision convolutional neural network (CNN) model into a low-precision version whose weights are constrained to be either powers of two or zero. Unlike existing methods which are struggled in noticeable accuracy loss, our INQ has the potential to resolve this issue, as benefiting from two innovations. On one hand, we introduce three interdependent operations, namely weight partition, group-wise quantization and re-training. A well-proven measure is employed to divide the weights in each layer of a pretrained CNN model into two disjoint groups. The weights in the first group are responsible to form a low-precision base, thus they are quantized by a variablelength encoding method. The weights in the other group are responsible to compensate for the accuracy loss from the quantization, thus they are the ones to be re-trained. On the other hand, these three operations are repeated on the latest re-trained group in an iterative manner until all the weights are converted into low-precision ones, acting as an incremental network quantization and accuracy enhancement procedure. Extensive experiments on the ImageNet classification task using almost all known deep CNN architectures including AlexNet, VGG-16, GoogleNet and ResNets well testify the efficacy of the proposed method. Specifically, at 5-bit quantization (a variable-length encoding: 1 bit for representing zero value, and the remaining 4 bits represent at most 16 different values for the powers of two) 1, our models have improved accuracy than the 32-bit floating-point references. Taking ResNet-18 as an example, we further show that our quantized models with 4-bit, 3-bit and 2-bit ternary weights have improved or very similar accuracy against its 32-bit floating-point baseline. Besides, impressive results with the combination of network pruning and INQ are also reported. We believe that our method sheds new insights on how to make deep CNNs to be applicable on mobile or embedded devices. The code will be made publicly available.
1 INTRODUCTION
Deep convolutional neural networks (CNNs) have demonstrated record breaking results on a variety of computer vision tasks such as image classification (Krizhevsky et al., 2012; Simonyan & Zisserman, 2015), face recognition (Taigman et al., 2014; Sun et al., 2014), semantic segmentation (Long et al., 2015; Chen et al., 2015a) and object detection (Girshick, 2015; Ren et al., 2015). Regardless of the availability of significantly improved training resources such as abundant annotated data, powerful computational platforms and diverse training frameworks, the promising results of deep CNNs are mainly attributed to the large number of learnable parameters, ranging from tens of millions to even hundreds of millions. Recent progress further shows clear evidence that CNNs could easily enjoy the accuracy gain from the increased network depth and width (He et al., 2016; Szegedy et al., 2015; 2016). However, this in turn lays heavy burdens on the memory and other
∗This work was done when Aojun Zhou was an intern at Intel Labs China, supervised by Anbang Yao who proposed the original idea and is responsible for correspondence. The first three authors contributed equally to the writing of the paper.
1This notation applies to our method throughout the paper.
computational resources. For instance, ResNet-152, a specific instance of the latest residual network architecture wining ImageNet classification challenge in 2015, has a model size of about 230 MB and needs to perform about 11.3 billion FLOPs to classify a 224× 224 image crop. Therefore, it is very challenging to deploy deep CNNs on the devices with limited computation and power budgets.
Substantial efforts have been made to the speed-up and compression on CNNs during training, feedforward test or both of them. Among existing methods, the category of network quantization methods attracts great attention from researches and developers. Some network quantization works try to compress pre-trained full-precision CNN models directly. Gong et al. (2014) address the storage problem of AlexNet (Krizhevsky et al., 2012) with vector quantization techniques. By replacing the weights in each of the three fully connected layers with respective floating-point centroid values obtained from the clustering, they can get over 20× model compression at about 1% loss in top-5 recognition rate. HashedNet (Chen et al., 2015b) uses a hash function to randomly map pre-trained weights into hash buckets, and all the weights in the same hash bucket are constrained to share a single floating-point value. In HashedNet, only the fully connected layers of several shallow CNN models are considered. For better compression, Han et al. (2016) present deep compression method which combines the pruning (Han et al., 2015), vector quantization and Huffman coding, and reduce the model storage by 35× on AlexNet and 49× on VGG-16 (Simonyan & Zisserman, 2015). Vanhoucke et al. (2011) use an SSE 8-bit fixed-point implementation to improve the computation of neural networks on the modern Intel x86 CPUs in feed-forward test, yielding 3× speed-up over an optimized floating-point baseline. Training CNNs by substituting the 32-bit floating-point representation with the 16-bit fixed-point representation has also been explored in Gupta et al. (2015). Other seminal works attempt to restrict CNNs into low-precision versions during training phase. Soudry et al. (2014) propose expectation backpropagation (EBP) to estimate the posterior distribution of deterministic network weights. With EBP, the network weights can be constrained to +1 and -1 during feed-forward test in a probabilistic way. BinaryConnect (Courbariaux et al., 2015) further extends the idea behind EBP to binarize network weights during training phase directly. It has two versions of network weights: floating-point and binary. The floating-point version is used as the reference for weight binarization. BinaryConnect achieves state-of-the-art accuracy using shallow CNNs for small datasets such as MNIST (LeCun et al., 1998) and CIFAR-10. Later on, a series of efforts have been invested to train CNNs with low-precision weights, low-precision activations and even low-precision gradients, including but not limited to BinaryNet (Courbariaux et al., 2016), XNOR-Net (Rastegari et al., 2016), ternary weight network (TWN) (Li & Liu, 2016), DoReFa-Net (Zhou et al., 2016) and quantized neural network (QNN) (Hubara et al., 2016).
Despite these tremendous advances, CNN quantization still remains an open problem due to two critical issues which have not been well resolved yet, especially under scenarios of using low-precision weights for quantization. The first issue is the non-negligible accuracy loss for CNN quantization methods, and the other issue is the increased number of training iterations for ensuring convergence. In this paper, we attempt to address these two issues by presenting a novel incremental network quantization (INQ) method.
In our INQ, there is no assumption on the CNN architecture, and its basic goal is to efficiently convert any pre-trained full-precision (i.e., 32-bit floating-point) CNN model into a low-precision version whose weights are constrained to be either powers of two or zero. The advantage of such kind of low-precision models is that the original floating-point multiplication operations can be replaced by cheaper binary bit shift operations on dedicated hardware like FPGA. We noticed that most existing network quantization methods adopt a global strategy in which all the weights are simultaneously converted to low-precision ones (that are usually in the floating-point types). That is, they have not considered the different importance of network weights, leaving the room to retain network accuracy limited. In sharp contrast to existing methods, our INQ makes a very careful handling for the model accuracy drop from network quantization. To be more specific, it incorporates three interdependent operations: weight partition, group-wise quantization and re-training. Weight partition uses a pruning-inspired measure (Han et al., 2015; Guo et al., 2016) to divide the weights in each layer of a pre-trained full-precision CNN model into two disjoint groups which play complementary roles in our INQ. The weights in the first group are quantized to be either powers of two or zero by a variable-length encoding method, forming a low-precision base for the original model. The weights in the other group are re-trained while keeping the quantized weights fixed, compensating for the accuracy loss resulted from the quantization. Furthermore, these three operations are repeated on the
latest re-trained weight group in an iterative manner until all the weights are quantized, acting as an incremental network quantization and accuracy enhancement procedure (as illustrated in Figure 1).
The main insight of our INQ is that a compact combination of the proposed weight partition, groupwise quantization and re-training operations has the potential to get a lossless low-precision CNN model from any full-precision reference. We conduct extensive experiments on the ImageNet large scale classification task using almost all known deep CNN architectures to validate the effectiveness of our method. We show that: (1) For AlexNet, VGG-16, GoogleNet and ResNets with 5-bit quantization, INQ achieves improved accuracy in comparison with their respective full-precision baselines. The absolute top-1 accuracy gain ranges from 0.13% to 2.28%, and the absolute top-5 accuracy gain is in the range of 0.23% to 1.65%. (2) INQ has the property of easy convergence in training. In general, re-training with less than 8 epochs could consistently generate a lossless model with 5-bit weights in the experiments. (3) Taking ResNet-18 as an example, our quantized models with 4-bit, 3-bit and 2-bit ternary weights also have improved or very similar accuracy compared with its 32-bit floating-point baseline. (4) Taking AlexNet as an example, the combination of our network pruning and INQ outperforms deep compression method (Han et al., 2016) with significant margins.
2 INCREMENTAL NETWORK QUANTIZATION
In this section, we clarify the insight of our INQ, describe its key components, and detail its implementation.
2.1 WEIGHT QUANTIZATION WITH VARIABLE-LENGTH ENCODING
Suppose a pre-trained full-precision (i.e., 32-bit floating-point) CNN model can be represented by {Wl : 1 ≤ l ≤ L}, where Wl denotes the weight set of the lth layer, and L denotes the number of learnable layers in the model. To simplify the explanation, we only consider convolutional layers and fully connected layers. For CNN models like AlexNet, VGG-16, GoogleNet and ResNets as tested in this paper, Wl can be a 4D tensor for the convolutional layer, or a 2D matrix for the fully connected layer. For simplicity, here the dimension difference is not considered in the expression. Given a pre-trained full-precision CNN model, the main goal of our INQ is to convert all 32-bit floating-point weights to be either powers of two or zero without loss of model accuracy. Besides, we also attempt to explore the limit of the expected bit-width under the premise of guaranteeing lossless network quantization. Here, we start with our basic network quantization method on how to
convert Wl to be a low-precision version Ŵl, and each of its entries is chosen from
Pl = {±2n1 , · · · ,±2n2 , 0}, (1)
where n1 and n2 are two integer numbers, and they satisfy n2 ≤ n1. Mathematically, n1 and n2 help to bound Pl in the sense that its non-zero elements are constrained to be in the range of either [−2n1 ,−2n2 ] or [2n2 , 2n1 ]. That is, network weights with absolute values smaller than 2n2 will be pruned away (i.e., set to zero) in the final low-precision model. Obviously, the problem is how to determine n1 and n2. In our INQ, the expected bit-width b for storing the indices in Pl is set beforehand, thus the only hyper-parameter shall be determined is n1 because n2 can be naturally computed once b and n1 are available. Here, n1 is calculated by using a tricky yet practically effective formula as
n1 = floor(log2(4s/3)), (2)
where floor(·) indicates the round down operation and s is calculated by using
s = max(abs(Wl)), (3)
where abs(·) is an element-wise operation and max(·) outputs the largest element of its input. In fact, Equation (2) helps to match the rounding power of 2 for s, and it could be easily implemented in practical programming. After n1 is obtained, n2 can be naturally determined as n2 = n1 + 1 − 2(b−1)/2. For instance, if b = 3 and n1 = −1, it is easy to get n2 = −2. Once Pl is determined, we further use the ladder of powers to convert every entry of Wl into a low-precision one by using
Ŵl(i, j) = { βsgn(Wl(i, j)) if (α+ β)/2 ≤ abs(Wl(i, j)) < 3β/2 0 otherwise,
(4)
where α and β are two adjacent elements in the sorted Pl, making the above equation as a numerical rounding to the quantum values. It should be emphasized that factor 4/3 in Equation (2) is set to make sure that all the elements in Pl correspond with the quantization rule defined in Equation (4). In other words, factor 4/3 in Equation (2) highly correlates with factor 3/2 in Equation (4).
Here, an important thing we want to clarify is the definition of the expected bit-width b. Taking 5-bit quantization as an example, since zero value cannot be written as the power of two, we use 1 bit to represent zero value, and the remaining 4 bits to represent at most 16 different values for the powers of two. That is, the number of candidate quantum values is at most 2b−1 + 1, so our quantization method actually adopts a variable-length encoding scheme. It is clear that the quantization described above is performed in a linear scale. An alternative solution is to perform the quantization in the log scale. Although it may also be effective, it should be a little bit more difficult in implementation and may cause some extra computational overhead in comparison to our method.
2.2 INCREMENTAL QUANTIZATION STRATEGY
We can naturally use the above described method to quantize any pre-trained full-precision CNN model. However, noticeable accuracy loss appeared in the experiments when using small bit-width values (e.g., 5-bit, 4-bit, 3-bit and 2-bit).
In the literature, there are many existing network quantization works such as HashedNet (Chen et al., 2015b), vector quantization (Gong et al., 2014), fixed-point representation (Vanhoucke et al., 2011; Gupta et al., 2015), BinaryConnect (Courbariaux et al., 2015), BinaryNet (Courbariaux et al., 2016), XNOR-Net (Rastegari et al., 2016), TWN (Li & Liu, 2016), DoReFa-Net (Zhou et al., 2016) and QNN (Hubara et al., 2016). Similar to our basic network quantization method, they also suffer from non-negligible accuracy loss on deep CNNs, especially when being applied on the ImageNet large scale classification dataset. For all these methods, a common fact is that they adopt a global strategy in which all the weights are simultaneously converted into low-precision ones, which in turn causes accuracy loss. Compared with the methods focusing on the pre-trained models, accuracy loss becomes worse for the methods such as XNOR-Net, TWN, DoReFa-Net and QNN which intend to train low-precision CNNs from scratch.
Recall that our main goal is to achieve lossless low-precision quantization for any pre-trained fullprecision CNN model with no assumption on its architecture. To this end, our INQ makes a special
handling of the strategy for suppressing resulting quantization loss in model accuracy. We are partially inspired by the latest progress in network pruning (Han et al., 2015; Guo et al., 2016). In these methods, the accuracy loss from removing less important network weights of a pre-trained neural network model could be well compensated by following re-training steps. Therefore, we conjecture that the nature of changing network weight importance is critical to achieve lossless network quantization.
Base on this assumption, we present INQ which incorporates three interdependent operations: weight partition, group-wise quantization and re-training. Weight partition is to divide the weights in each layer of a pre-trained full-precision CNN model into two disjoint groups which play complementary roles in our INQ. The weights in the first group are responsible for forming a low-precision base for the original model, thus they are quantized by using Equation (4). The weights in the second group adapt to compensate for the loss in model accuracy, thus they are the ones to be re-trained. Once the first run of the quantization and re-training operations is finished, all the three operations are further conducted on the second weight group in an iterative manner, until all the weights are converted to be either powers of two or zero, acting as an incremental network quantization and accuracy enhancement procedure. As a result, accuracy loss under low-precision CNN quantization can be well suppressed by our INQ. Illustrative results at iterative steps of our INQ are provided in Figure 2.
For the lth layer, weight partition can be defined as
A (1) l ∪A (2) l = {Wl(i, j)}, and A (1) l ∩A (2) l = ∅, (5)
where A(1)l denotes the first weight group that needs to be quantized, and A2 denotes the other weight group that needs to be re-trained. We leave the strategies for group partition to be chosen in the experiment section. Here, we define a binary matrix Tl to help distinguish above two categories of weights. That is, Tl(i, j) = 0 means Wl(i, j) ∈ A(1)l , and Tl(i, j) = 1 means Wl(i, j) ∈ A(2)l .
2.3 INCREMENTAL NETWORK QUANTIZATION ALGORITHM
Now, we come to the training method. Taking the lth layer as an example, the basic optimization problem of making its weights to be either powers of two or zero can be expressed as
min Wl E(Wl) = L(Wl) + λR(Wl)
s.t. Wl(i, j) ∈ Pl, 1 ≤ l ≤ L, (6)
where L(Wl) is the network loss, R(Wl) is the regularization term, λ is a positive coefficient, and the constraint term indicates each weight entry Wl(i, j) should be chosen from the set Pl consisting of a fixed number of the values of powers of two plus zero. Direct solving above optimization problem in training from scratch is challenging since it is very easy to undergo convergence problem.
By performing weight partition and group-wise quantization operations beforehand, the optimization problem defined in (6) can be reshaped into a easier version. That is, we only need to optimize the following objective function
min Wl E(Wl) = L(Wl) + λR(Wl)
s.t. Wl(i, j) ∈ Pl, if Tl(i, j) = 0, 1 ≤ l ≤ L, (7)
where Pl is determined at group-wise quantization operation, and the binary matrix Tl acts as a mask which is determined by weight partition operation. Since Pl and Tl are known, the optimization problem (7) can be solved using popular stochastic gradient decent (SGD) method. That is, in INQ, we can get the update scheme for the re-training as
Wl(i, j)←Wl(i, j)− γ ∂E
∂(Wl(i, j)) Tl(i, j), (8)
where γ is a positive learning rate. Note that the binary matrix Tl forces zero update to the weights that have been quantized. That is, only the weights still keep with floating-point values are updated, akin to the latest pruning methods (Han et al., 2015; Guo et al., 2016) in which only the weights that are not currently removed are re-trained to enhance network accuracy. The whole procedure of our INQ is summarized as Algorithm 1.
We would like to highlight that the merits of our INQ are in three aspects: (1) Weight partition introduces the importance-aware weight quantization. (2) Group-wise weight quantization introduces much less accuracy loss than simultaneously quantizing all the network weights, thus making retraining have larger room to recover model accuracy. (3) By integrating the operations of weight partition, group-wise quantization and re-training into a nested loop, our INQ has the potential to obtain lossless low-precision CNN model from the pre-trained full-precision reference.
Algorithm 1 Incremental network quantization for lossless CNNs with low-precision weights. Input: X: the training data, {Wl : 1 ≤ l ≤ L}: the pre-trained full-precision CNN model, {σ1, σ2, · · · , σN}: the accumulated portions of weights quantized at iterative steps Output: {Ŵl : 1 ≤ l ≤ L}: the final low-precision model with the weights constrained to be either powers of two or zero
1: Initialize A(1)l ← ∅, A (2) l ← {Wl(i, j)}, Tl ← 1, for 1 ≤ l ≤ L 2: for n = 1, 2, . . . , N do 3: Reset the base learning rate and the learning policy 4: According to σn, perform layer-wise weight partition and update A (1) l , A (2) l and Tl 5: Based on A(1)l , determine Pl layer-wisely 6: Quantize the weights in A(1)l by Equation (4) layer-wisely 7: Calculate feed-forward loss, and update weights in {A(2)l : 1 ≤ l ≤ L} by Equation (8) 8: end for
3 EXPERIMENTAL RESULTS
To analyze the performance of our INQ, we perform extensive experiments on the ImageNet large scale classification task, which is known as the most challenging image classification benchmark so far. ImageNet dataset has about 1.2 million training images and 50 thousand validation images. Each image is annotated as one of 1000 object classes. We apply our INQ to AlexNet, VGG-16, GoogleNet, ResNet-18 and ResNet-50, covering almost all known deep CNN architectures. Using the center crops of validation images, we report the results with two standard measures: top-1 error rate and top-5 error rate. For fair comparison, all pre-trained full-precision (i.e., 32-bit floatingpoint) CNN models except ResNet-18 are taken from the Caffe model zoo2. Note that He et al. (2016) do not release their pre-trained ResNet-18 model to the public, so we use a publicly available re-implementation by Facebook3. Since our method is implemented with Caffe, we make use of an open source tool4 to convert the pre-trained ResNet-18 model from Torch to Caffe.
3.1 RESULTS ON IMAGENET
Setting expected bit-width to 5, the first set of experiments is performed to testify the efficacy of our INQ on different CNN architectures. Regarding weight partition, there are several candidate strategies as we tried in our previous work for efficient network pruning (Guo et al., 2016). In Guo et al. (2016), we found random partition and pruning-inspired partition are the two best choices compared with the others. Thus in this paper, we directly compare these two strategies for weight partition. In random strategy, the weights in each layer of any pre-trained full-precision deep CNN model are randomly split into two disjoint groups. In pruning-inspired strategy, the weights are divided into two disjoint groups by comparing their absolute values with layer-wise thresholds which are automatically determined by a given splitting ratio. Here we directly use pruning-inspired strategy and the experimental results in Section 3.2 will show why. After the re-training with no more than 8 epochs over each pre-trained full-precision model, we obtain the results as shown in Table 1. It can be concluded that the 5-bit CNN models generated by our INQ show consistently improved top-1 and top-5 recognition rates compared with respective full-precision references. Parameter settings are described below.
AlexNet: AlexNet has 5 convolutional layers and 3 fully-connected layers. We set the accumulated portions of quantized weights at iterative steps as {0.3, 0.6, 0.8, 1}, the batch size as 256, the weight decay as 0.0005, and the momentum as 0.9.
VGG-16: Compared with AlexNet, VGG-16 has 13 convolutional layers and more parameters. We set the accumulated portions of quantized weights at iterative steps as {0.5, 0.75, 0.875, 1}, the batch size as 32, the weight decay as 0.0005, and the momentum as 0.9.
2https://github.com/BVLC/caffe/wiki/Model-Zoo 3https://github.com/facebook/fb.resnet.torch/tree/master/pretrained 4https://github.com/zhanghang1989/fb-caffe-exts
GoogleNet: Compared with AlexNet and VGG-16, GoogleNet is more difficult to quantize due to a smaller number of parameters and the increased network width. We set the accumulated portions of quantized weights at iterative steps as {0.2, 0.4, 0.6, 0.8, 1}, the batch size as 80, the weight decay as 0.0002, and the momentum as 0.9.
ResNet-18: Different from above three networks, ResNets have batch normalization layers and relief the vanishing gradient problem by using shortcut connections. We first test the 18-layer version for exploratory purpose and test the 50-layer version later on. The network architectures of ResNet18 and ResNet-34 are very similar. The only difference is the number of filters in every convolutional layer. We set the accumulated portions of quantized weights at iterative steps as {0.5, 0.75, 0.875, 1}, the batch size as 80, the weight decay as 0.0005, and the momentum as 0.9. ResNet-50: Besides significantly increased network depth, ResNet-50 has a more complex network architecture in comparison to ResNet-18. However, regarding network architecture, ResNet-50 is very similar to ResNet-101 and ResNet-152. The only difference is the number of filters in every convolutional layer. We set the accumulated portions of quantized weights at iterative steps as {0.5, 0.75, 0.875, 1}, the batch size as 32, the weight decay as 0.0005, and the momentum as 0.9.
3.2 ANALYSIS OF WEIGHT PARTITION STRATEGIES
In our INQ, the first operation is weight partition whose result will directly affect the following group-wise quantization and re-training operations. Therefore, the second set of experiments is conducted to analyze two candidate strategies for weight partition. As mentioned in the previous section, we use pruning-inspired strategy for weight partition. Unlike random strategy in which all the weights have equal probability to fall into the two disjoint groups, pruning-inspired strategy considers that the weights with larger absolute values are more important than the smaller ones to form a low-precision base for the original CNN model. We use ResNet-18 as a test case to compare the performance of these two strategies. In the experiments, the parameter settings are completely the same as described in Section 3.1. We set 4 epochs for weight re-training. Table 2 summarizes the results of our INQ with 5-bit quantization. It can be seen that our INQ achieves top-1 error rate of 32.11% and top-5 error rate of 11.73% by using random partition. Comparatively, pruning-inspired partition brings 1.09% and 0.83% decrease in top-1 and top-5 error rates, respectively. Apparently, pruning-inspired partition is better than random partition, and this is the reason why we use it in this paper. For future works, weight partition based on quantization error could also be an option worth exploring.
3.3 THE TRADE-OFF BETWEEN EXPECTED BIT-WIDTH AND MODEL ACCURACY
The third set of experiments is performed to explore the limit of the expected bit-width under which our INQ can still achieve lossless network quantization. Similar to the second set of experiments, we also use ResNet-18 as a test case, and the parameter settings for the batch size, the weight decay and the momentum are completely the same. Finally, lower-precision models with 4-bit, 3-bit and even 2-bit ternary weights are generated for comparisons. As the expected bit-width goes down, the number of candidate quantum values will be decreased significantly, thus we shall increase the number of iterative steps accordingly for enhancing the accuracy of final low-precision model. Specifically, we set the accumulated portions of quantized weights at iterative steps as {0.3, 0.5, 0.8, 0.9, 0.95, 1}, {0.2, 0.4, 0.6, 0.7, 0.8, 0.9, 0.95, 1} and {0.2, 0.4, 0.6, 0.7, 0.8, 0.85, 0.9, 0.95, 0.975, 1} for 4-bit, 3-bit and 2-bit ternary models, respectively. The required number of epochs also increases when the expected bit-width goes down, and it reaches 30 when training our 2-bit ternary model. Although our 4-bit model shows slightly decreased accuracy when compared with the 5-bit model, its accuracy is still better than that of the pre-trained full-precision model. Comparatively, even when the expected bit-width goes down to 3, our low-precision model shows only 0.19% and
0.33% losses in top-1 and top-5 recognition rates, respectively. As for our 2-bit ternary model, although it incurs 2.25% decrease in top-1 error rate and 1.56% decrease in top-5 error rate in comparison to the pre-trained full-precision reference, its accuracy is considerably better than stateof-the-art results reported for binary-weight network (BWN) (Rastegari et al., 2016) and ternary weight network (TWN) (Li & Liu, 2016). Detailed results are summarized in Table 3 and Table 4.
3.4 LOW-BIT DEEP COMPRESSION
In the literature, recently proposed deep compression method (Han et al., 2016) reports so far best results on network compression without loss of model accuracy. Therefore, the last set of experiments is conducted to explore the potential of our INQ for much better deep compression. Note that Han et al. (2016) is a hybrid network compression solution combining three different techniques, namely network pruning (Han et al., 2015), vector quantization (Gong et al., 2014) and Huffman coding. Taking AlexNet as an example, network pruning gets 9× compression, however this result is mainly obtained from the fully connected layers. Actually its compression performance on the convolutional layers is less than 3× (as can be seen in the Table 4 of Han et al. (2016)). Besides, network pruning is realized by separately performing pruning and re-training in an iterative way, which is very time-consuming. It will cost at least several weeks for compressing AlexNet. We solved this problem by our dynamic network surgery (DNS) method (Guo et al., 2016) which achieves about 7× speed-up in training and improves the performance of network pruning from 9× to 17.7×. In Han et al. (2016), after network pruning, vector quantization further improves compression ratio from 9× to 27×, and Huffman coding finally boosts compression ratio up to 35×. For fair comparison, we combine our proposed INQ and DNS, and compare the resulting method with Han et al. (2016). Detailed results are summarized in Table 5. When combing our proposed INQ and DNS, we achieve much better compression results compared with Han et al. (2016). Specifically, with 5-bit quantization, we can achieve 53× compression with slightly larger gains both in top-5 and top-1 recognition rates, yielding 51.43%/96.30% absolute improvement in compression performance compared with full version/fair version (i.e., the combination of network pruning and vector quantization) of Han et al. (2016), respectively. Consistently better results have also obtained for our 4-bit and 3-bit models.
Besides, we also perform a set of experiments on AlexNet to compare the performance of our INQ and vector quantization (Gong et al., 2014). For fair comparison, re-training is also used to enhance the performance of vector quantization, and we set the number of cluster centers for all of 5 convolutional layers and 3 fully connect layers to 32 (i.e., 5-bit quantization). In the experiment, vector quantization incurs over 3% loss in model accuracy. When we change the number of cluster centers for convolutional layers from 32 to 128, it gets an accuracy loss of 0.98%. This is consistent with the results reported in (Gong et al., 2014). Comparatively, vector quantization is mainly proposed
to compress the parameters in the fully connected layers of a pre-trained full-precision CNN model, while our INQ addresses all network layers simultaneously and has no accuracy loss for 5-bit and 4-bit quantization. Therefore, it is evident that our INQ is much better than vector quantization. Last but not least, the final weights for vector quantization (Gong et al., 2014), network pruning (Han et al., 2015) and deep compression (Han et al., 2016) are still floating-point values, but the final weights for our INQ are in the form of either powers of two or zero. The direct advantage of our INQ is that the original floating-point multiplication operations can be replaced by cheaper binary bit shift operations on dedicated hardware like FPGA.
4 CONCLUSIONS
In this paper, we present INQ, a new network quantization method, to address the problem of how to convert any pre-trained full-precision (i.e., 32-bit floating-point) CNN model into a lossless lowprecision version whose weights are constrained to be either powers of two or zero. Unlike existing methods which usually quantize all the network weights simultaneously, INQ is a more compact quantization framework. It incorporates three interdependent operations: weight partition, groupwise quantization and re-training. Weight partition splits the weights in each layer of a pre-trained full-precision CNN model into two disjoint groups which play complementary roles in INQ. The weights in the first group is directly quantized by a variable-length encoding method, forming a low-precision base for the original CNN model. The weights in the other group are re-trained while keeping all the quantized weights fixed, compensating for the accuracy loss from network quantization. More importantly, the operations of weight partition, group-wise quantization and re-training are repeated on the latest re-trained weight group in an iterative manner until all the weights are quantized, acting as an incremental network quantization and accuracy enhancement procedure. On the ImageNet large scale classification task, we conduct extensive experiments and show that our quantized CNN models with 5-bit, 4-bit, 3-bit and even 2-bit ternary weights have improved or at least comparable accuracy against their full-precision baselines, including AlexNet, VGG-16, GoogleNet and ResNets. As for future works, we plan to extend incremental idea behind INQ from low-precision weights to low-precision activations and low-precision gradients (we have actually already made some good progress on it, as shown in our supplementary materials). We will also investigate computation and power efficiency by implementing our low-precision CNN models on hardware platforms.
A APPENDIX 1: STATISTICAL ANALYSIS OF THE QUANTIZED WEIGHTS
Taking our 5-bit AlexNet model as an example, we analyze the distribution of the quantized weights. Detailed statistical results are summarized in Table 6. We can find: (1) in the 1st and 2nd convolutional layers, the values of {−2−6, −2−5, −2−4, 2−6, 2−5, 2−4} and {−2−8, −2−7, −2−6, −2−5, 0, 2−8, 2−7, 2−6, 2−5} occupy over 60% and 94% of all quantized weights, respectively; (2) the distributions of the quantized weights in the 3rd, 4th and 5th convolutional layers are similar to that of the 2nd convolutional layer, and more weights are quantized into zero in the 2nd, 3rd, 4th and 5th convolutional layers compared with the 1st convolutional layer; (3) in the 1st fully connected layer, the values of {−2−10, −2−9, −2−8, −2−7, 0, 2−10, 2−9, 2−8, 2−7} occupy about 98% of all quantized weights, and similar results can be seen for the 2nd fully connected layer; (4) generally, the distributions of the quantized weights in the convolutional layers are usually more scattered compared with the fully connected layers. This may be partially the reason why it is much easier to get good compression performance on fully connected layers in comparison to convolutional layers, when using methods such as network hashing (Chen et al., 2015b) and vector quantization (Gong et al., 2014); (5) for 5-bit AlexNet model, the required bit-width for each layer is actually 4 but not 5.
B APPENDIX 2: LOSSLESS CNNS WITH LOW-PRECISION WEIGHTS AND LOW-PRECISION ACTIVATIONS
Recently, we have made some good progress on developing our INQ for lossless CNNs with both low-precision weights and low-precision activations. According to the results summarized in Table 7, it can be seen that our VGG-16 model with 5-bit weights and 4-bit activations shows improved top-5 and top-1 recognition rates in comparison to the pre-trained reference with 32-bit floating-point weights and 32-bit floating-point activations. To the best of our knowledge, this should be the best results reported on VGG-16 architecture so far.

There is a great deal of ongoing interest in compressing neural network models. One line of work has focused on using low-precision representations of the model weights, even down to 1 or 2 bits. However, so far these approaches have been accompanied by a significant impact on accuracy. The paper proposes an iterative quantization scheme, in which the network weights are quantized in stages---the largest weights (in absolute value) are quantized and fixed, while unquantized weights can adapt to compensate for any resulting error. The experimental results show this is extremely effective, yielding models with 4 bit or 3 bit weights with essentially no reduction in accuracy. While at 2 bits the accuracy decreases slightly, the results are substantially better than those achieved with other quantization approaches.
Overall this paper is clear, the technique is as far as I am aware novel, the experiments are thorough and the results are very compelling, so I recommend acceptance. The paper could use another second pass for writing style and grammar. Also, the description of the pruning-inspired partitioning strategy could be clarified somewhat... e.g., the chosen splitting ratio of 50% only seems to be referenced in a figure caption and not the main text.


LAYER RECURRENT NEURAL NETWORKS
In this paper, we propose a Layer-RNN (L-RNN) module that is able to learn contextual information adaptively using within-layer recurrence. Our contributions are three-fold: (i) we propose a hybrid neural network architecture that interleaves traditional convolutional layers with L-RNN module for learning longrange dependencies at multiple levels; (ii) we show that a L-RNN module can be seamlessly inserted into any convolutional layer of a pre-trained CNN, and the entire network then fine-tuned, leading to a boost in performance; (iii) we report experiments on the CIFAR-10 classification task, showing that a network with interleaved convolutional layers and L-RNN modules, achieves comparable results (5.39% top1 error) using only 15 layers and fewer parameters to ResNet-164 (5.46%); and on the PASCAL VOC2012 semantic segmentation task, we show that the performance of a pre-trained FCN network can be boosted by 5% (mean IOU) by simply inserting Layer-RNNs.
In this paper, we propose a Layer-RNN (L-RNN) module that is able to learn contextual information adaptively using within-layer recurrence. Our contributions are three-fold: (i) we propose a hybrid neural network architecture that interleaves traditional convolutional layers with L-RNN module for learning longrange dependencies at multiple levels; (ii) we show that a L-RNN module can be seamlessly inserted into any convolutional layer of a pre-trained CNN, and the entire network then fine-tuned, leading to a boost in performance; (iii) we report experiments on the CIFAR-10 classification task, showing that a network with interleaved convolutional layers and L-RNN modules, achieves comparable results (5.39% top1 error) using only 15 layers and fewer parameters to ResNet-164 (5.46%); and on the PASCAL VOC2012 semantic segmentation task, we show that the performance of a pre-trained FCN network can be boosted by 5% (mean IOU) by simply inserting Layer-RNNs.
1 INTRODUCTION
In computer vision tasks, such as image classification or pixel level prediction, multi-scale contextual information plays a very important role in achieving high performance. The original architectures for these tasks (e.g. He et al. (2016a); Krizhevsky et al. (2012); Long et al. (2015); Ronneberger et al. (2015); Simonyan & Zisserman (2015); Szegedy et al. (2015)) were able to obtain multi-scale context with a large spatial footprint by the combination of filters through the layers of the network, so that a large receptive field was effectively built up. Indeed, the final layers of these networks use average pooling or fully connected layers (convolution with a large kernel) so that the effective receptive field covers the entire input image patch. More recent pixel prediction architectures have used dilated convolutions (Yu & Koltun, 2016; Chen et al., 2016) which are able to aggregate multi-scale contextual information without losing resolution (due to the spatial pooling and strides in the original architectures), and without incurring the penalty of having to learn many parameters for convolutions with very large kernels.
In this paper we introduce an alternative ‘module’ for learning multi-scale spatial contextual information by using Recurrent Neural Networks (RNNs) within layers. This approach is inspired by the ReNet architecture of Visin et al. (2015), which we extend here into a hybrid architecture that interleaves traditional convolutional neural network (CNN) modules with layer recurrent modules, and we term a Layer Recurrent Neural Network (L-RNN). A L-RNN module is a combination of 1D RNNs, and is able to learn contextual information adaptively, with the effective receptive field able to reach across the entire feature map or image, if that is required for the task. The hybrid network combines the best of both worlds: canonical CNNs are composed of filters that are efficient in capturing features in a local region, whilst the L-RNNs are able to learn long-range dependencies across a layer efficiently with only a small number of parameters.
We describe the basic L-RNN module in Section 2, and discuss different fusion choices for the hybrid architecture by incorporating L-RNN into residual blocks (He et al., 2016b) in Section 3. In addition, in Section 4, we explain how L-RNN modules can be inserted into pre-trained CNNs seamlessly. This means that the entire network does not have to be trained from scratch, only the added L-RNNs are fine-tuned together with pre-trained networks, and the experiments show that this addition always improves performance. In Section 5, we experiment on the CIFAR-10 classification with the hybrid networks of increasing depths, by using Layer Normalization (Ba et al., 2016), we are able to train vanilla RNNs to match the performance of GRU (Chung et al., 2015), while
using fewer parameters. In addition, we fine-tune a truncated VGG-16 FCN base net for semantic segmentation on the Pascal VOC 2012 and COCO (Lin et al., 2014) dataset.
It is worth noting that (broadly) recurrence can be used in feed-forward multi-layer convolutional neural network architectures in two ways: between layers, and within layers. For example, betweenlayer recurrence was used for scene labelling in (Liang et al., 2015; Pinheiro & Collobert, 2014) with convolutions applied recursively on top of feature maps from different layers or raw input images. And in (Zheng et al., 2015), spatial dependencies are modelled explicitly for semantic segmentation with densely connected Gaussian CRFs by iterated application of bilateral filtering using betweenlayer recurrence.
By contrast, our Layer-RNN architecture falls into the second category, where within-layer recurrence is used to capture dependencies. Others have learnt contextual information from within layer recurrence for tasks such as object detection (Bell et al., 2016), and low-level vision problems, such as de-noising, colourization and smoothing (Liu et al., 2016). We postpone discussing in detail the relationships of the proposed Layer-RNN modules to these architectures, and to that of ReNet (Visin et al., 2015) and ReSeg (Visin et al., 2016), until we have introduced the L-RNN in Section 2.
2 LAYER-RNN ARCHITECTURE
The architecture of the network (Figure 1) is composed of two parts. Local features are calculated by the low-level CNNs module, the Layer-RNN (L-RNN) module, consisting of several 1D spatial RNNs is applied to capture the spatial dependencies. By scanning across the feature maps in different directions, the complete L-RNN is able to learn the receptive field in an adaptive way, up to the size of the entire image. These two modules can be combined to build networks in various ways; for example, an L-RNN module can be stacked on top of several CNN modules at the final layer, or CNN and L-RNN modules can be interleaved at multiple levels.
2.1 LAYER-RNN MODULE
As shown in Figure 1, the Layer-RNN (L-RNN) module is a combination of the 1D spatial recurrent modules (B) and (C). In each module, there are two 1D RNNs scanning across the feature maps horizontally or vertically from two directions (bidirectional spatial RNNs), and their hidden states are updated at every spatial step. Consequently, for each of the horizontal and vertical directions, two output feature maps are obtained with the same width and height as the input feature maps. In our implementation, we simply sum up these output feature maps (an alternative is to concatenate the output feature maps, but that would increase the number of parameters).
More formally, assume the feature maps (layer L) coming into the L-RNN module are XL P Rm n d and output XL 1 (layer L 1), where m,n, d refers to the width, height, and the number of feature maps respectively for the input layer. For simplicity, assume the input to the 1D spatial
RNNs from XL is a feature vector at each spatial location, each row or column on the feature maps is treated as one sequence. When scanning from left to right, the feature responses for location ij can be calculated:
xL 1i,j fpUx L i,j V x L 1 i,j 1 bq left to right (1)
Where xL 1i,0 0, x L i,j P Rd 1, x L 1 i,j , x L 1 i,j 1 P RD 1, U P RD d, V P RD D, b P RD 1, D denotes the number of nodes used in the 1D spatial RNN, and f refers to the non-linearity function. 1D spatial RNNs scanning other directions can be calculated similarly. Notice that, the first term of equation 1 encodes local information independently, resembling the normal convolutional layer, and the second term characterizes the within-layer recurrence (U is a convolution matrix, V a recurrence matrix). We make use of this observation in Section 4.
2.2 DISCUSSION AND RELATION TO OTHER WORK
As can be seen in Figure 1C, the effective receptive field can cover the entire image. However, the actual receptive field depends on the parameters of the RNNs, and can be learnt adaptively. As an insight to what is learnt, consider a separable filter, such as an axis aligned 2D Gaussian. Such filters can be applied exactly by a composition of 1D Gaussian convolutions in the horizontal and vertical directions. The 1D spatial RNNs can approximate finite 1D convolutions of this type.
We next discuss the relation of the L-RNN to prior work. First, ReNets (Visin et al., 2015), which is an architecture completely made of 1D RNNs (i.e. no CNNs). In ReNets, the input images are first split into non-overlapping patches of size m n d, where m,n, d refer to width, height and feature channels respectively. The 1D RNNs takes the flattened patch (mn d) as input, and outputs feature vector of size D 1, where D refers to the number of nodes used in the RNNs. In contrast, we interleave the L-RNN and CNN modules. There are two benefits of this: first, CNNs are more efficient at capturing local features than RNNs, the L-RNN stacked upon them is able to learn dependencies between local features (rather than the input channel reformatted); second, we are able to introduce more non-linearities between the hierarchical layers (through the convolutional+ReLU and pooling layers), and a RNN provides non-linearities within the same layer.
The 2D-RNN, proposed in (Graves & Schmidhuber, 2009; Theis & Bethge, 2015), is able to scan across the image or feature maps row-by-row, or column-by-column sequentially, with each RNN node accept input from three sources, namely, projections of current input, and feedbacks from the two neighbour nodes. By contrast, we use unidirectional 1D spatial RNNs, with each hidden node only accepting feedbacks from its previous node. Another advantage of our model is that rows or columns can be processed in parallel on GPUs, and training time is shortened.
Bell et al. (2016) (Inside-Outside Net) and Visin et al. (2016) (ReSeg) describe similar ideas for object detection and semantic segmentation. Both architectures follow a pipeline that consists of a CNN feature extractor (VGG Net) followed by spatial RNNs at the final prediction stage. In contrast, we treat the L-RNN module as a general computational layer, that can be inserted into any layer of modern architectures, and interleaved with CNN modules. This enables a network to be capable of learning contextual information in a flexible way at multiple levels, rather than with hand-crafted kernel sizes and receptive fields.
Note that the vanilla RNN unit consists of two terms, a local term and a recurrence term, where the local term is exactly the convolution operation. Therefore, the spatial RNN can be seen as a generalisation of the convolutional layer, and in the worst case, when the RNN learns no context, the layer simply becomes a convolutional one. For tasks with limited data (semantic segmentation in our case), we propose a regime for inserting the L-RNN into the pre-trained FCN and fine-tuning the entire network end-to-end. This means that we directly increase the representational power of the model, and set the pre-trained model free to learn contextual information if it is needed.
3 CNNS & LAYER-RNN MODULES
In this section, we describe the architecture for incorporating 1D spatial RNNs into the computational block of a Residual Networks(He et al., 2016b), and also discuss fusion methods for such blocks.
We start with the standard residual block of He et al. (2016b) (Figure 2(a)), and then replace the included CNN layer with bidirectional spatial RNNs, to includ a L-RNN module instead.
We consider three fusion options for combining the features from such blocks with the input to subsequent layers; namely forward, sum and concatenation. Forward refers to the traditional feedforward architectures:
XL 1 F pXL,W q (2)
i.e. the block simply becomes a new layer; sum denotes the method of the original residual networks:
XL 1 XL F pXL,W q (3)
so that the L-RNN module acts as a residual block; whilst, in concatenation, features from multiple layers (same spatial sizes) are concatenated:
XL 1 rXL;F pXL,W qs (;) refers to concatenation (4)
Therefore, the channels of output feature maps will be the sum of the channels of the two concatenated layers (the number of parameters will be increased for the next layers). In the experimental evaluation of Section 5.1 we compare these options.
4 ADDING A LAYER-RNN TO A PRE-TRAINED CNN
In this section, we describe how a Layer-RNN module, can be seamlessly inserted into a pre-trained CNN. In a typical scenario, the CNN would be trained for classification on ImageNet (where there are copious annotations). After inserting the L-RNN modules, the hybrid L-RNN network can then be fine tuned for a new task such as pixel-level prediction, e.g. semantic segmentation (where the annotated data is usually more limited). This trick naturally allows multi-level contextual information to be effortlessly incorporated. Avoiding training the network from scratch means the entire network can be re-purposed with the available annotations and trained end-to-end for the new task, whilst benefiting from the earlier classification training.
We illustrate the idea using 1D convolution, but the same principles hold for the entire L-RNN module. As shown in Figure 3, the canonical CNN architecture for a 1D convolution can be denoted as:
XL 1 fpW XL bq (5)
where refers to convolution, W and b are the parameters of the CNN, L,L 1 denote the layer. The 1D spatial RNN can be written as :
XL 1i fpU X L i V X L 1 i 1 bq (6)
where U, V, b refer to the parameters that are shared across the whole scan-line. Notice that the 1D spatial RNN are designed to incorporate two terms, projections from local region (input-to-hidden) and recurrence term from previous hidden unit (hidden-to-hidden). In fact, it is
the presence of non-zero recurrence matrix V , that characterizes the 1D spatial RNN, and they can be calculated in a two-step way as :
Xinter U XL b ( Convolution ) (7)
XL 1i fpX inter i q ( i = 1, zero initial states ) (8)
XL 1i fpX inter i V X L 1 i 1 q pi ¡ 1q (9)
By interpreting the recurrence in this way, 1D spatial RNNs can be constructed by inserting recurrence directly into any convolutional layer right after the convolution. If the recurrence matrix V is initialized as zero, and ReLU is the activation function, then the 1D spatial RNN will be initialized exactly as the pre-trained CNNs. The complete L-RNN can be constructed by inserting two bidirectional spatial RNNs into subsequent layers of the pre-trained CNNs. We derive the expression of the within-layer gradient for use in back-prop fine-tuning in Appendix B.
5 EXPERIMENTAL EVALUATION
We test the proposed Layer-RNN on two supervised learning tasks: CIFAR-10 classification in Section 5.1; and PASCAL VOC 2012 segmentation in Section 5.2.
5.1 IMAGE CLASSIFICATION
In this section, we investigate classification performance under variations in an architecture containing L-RNN modules. We vary the depth of the network, the number and position of the L-RNN modules, the type of recurrent units in RNNs, the pooling mechanisms for the last pooling layer, and the method of fusing the block outputs.
Architectures: An overview of the architectures is given in Table 1 (with the fundamental building modules (CNN and L-RNN) adapted from Figure 2).
There are two principal architectural variations. The first variation is that from Network A to D, we gradually increase the network depth by adding CNN Modules, with the L-RNN module always stacked at the final stage to capture global information over the entire image, in a similar manner to the fully connected layers or average pooling in other networks. Network A has 5 convolutional layers.
The second principal variation, in Network E and F, is to interleave CNN and L-RNN modules. This means that the network is capable of learning representations across large spatial footprints at any stage in the network. To show the effectiveness of adding L-RNN modules, we include a Baseline-CNN composed of only convolutional layers (7 layers, with concatenation used at every skip layer). Network E is built upon the Baseline-CNN by inserting L-RNN modules before CNN modules at multiple stages. To make sure the performance gain is not from the increased number of parameters, we cut down the number of filters in the last CNN module to 128 (this number is 256 in the Baseline-CNN). Network F, uses more convolutional layers interleaved with L-RNN modules.
Other variations of architectures include: firstly, we may use Forward, Sum, Concatenation to fuse features; secondly, GRU and vanilla RNN units are compared for the L-RNN modules, ReLU is used for both cases as the non-linear activation; thirdly, both max pooling and average pooling are tested as global pooling. For clarity, we name the networks by these variations in Table 2. When Forward is selected to fuse features, Network A-Forward simply follows the traditional CNN with pure feed-forward layers. A-Concat uses concatenation as an alternative, and A-Sum follows the idea of residual networks proposed in (He et al., 2016b), the number of filters is gradually increased as the networks get deeper. To match dimensions for summation, 1 1 convolution is used in A-Sum. In our experiments, we found that concatenation works better than sum (Table 2). Therefore, in all
other architectures (B,C,D), as we gradually increase the network depth by adding CNN modules, we fuse the skip layers by only alternating between concatenation and forward.
Following the VGG-net (Simonyan & Zisserman, 2015), in all architectures, convolutional kernels in the CNN Module are of size 3 3. Maxpoolings (2 2) are used as intermediate pooling, and 8 8 global poolings (average or max) are applied at the end. To avoid overfitting, we use dropout (0.5). Training details and recurrent units are described in the Appendix A. Implementations are mostly based in Theano (Theano Development Team, 2016) with single NVIDIA Titan X.
Dataset & Evaluation. We conducted experiments on the CIFAR-10 dataset, which consists of 40k training images, 10k validation and 10k testing images in 10 classes, and each of the image is of 32 32 pixels with RGB channels. We augment the training data with simple transformations (rotation, flipping, scaling) on the fly. The mean image over the whole training set is subtracted from each image during training. Following the standard evaluation protocol, we report the top1 error on the testing set.
Results & Discussion. We present detailed comparisons with other published methods in Table 2.
From the experimental results, we can draw the following conclusions:
Comparison of basic choices. Max pooling consistently performs better when used as the global pooling in our case, this is seen in the results by Baseline-CNN-Avg (9.07%) vs. Baseline-CNNMax (8.48%), and A-Concat-RNN-Avg (7.65%) vs. A-Concat-RNN-Max (7.43%). One possible explanation would be that for classification tasks, decisions are based on the most salient features.
In our experiments for shallow networks, the summing of residual connections shows no benefit compared to feed-forward or concatenation. This observation is made from the results by AForward-GRU-Max (7.57%), A-Concat-GRU-Max (7.35%) and A-Sum-GRU-Max (7.69%). Thus, as also employed in U-Net or DenseNet (Ronneberger et al., 2015; Huang et al., 2016), concatenation can be used as an alternative to summation in building deeper networks.
It can be seen that vanilla RNN units trained with Layer Normalization (Ba et al., 2016) can perform almost as well as GRU, while saving a a large number of parameters (by comparing the results from A-Concat-RNN-Max with 0.9M parameters (7.43%) and that of A-Concat-GRU-Max with 1.95M parameters (7.36%), B-RNN-Max with 1.27M parameters (6.78%) vs. B-GRU-Max with 2.3M parameters (6.62%)).
Networks with L-RNN module stacked at the final stage. Even shallow networks with LRNN modules (architectures A) can achieve comparable or superior performance to deep architectures with 19 layers that requires more parameters (e.g. Network A-Concat-RNN-Max (0.9M ) vs. Highway(2.3M )). This confirms that when a L-RNN module is stacked on top of CNNs, it is able to capture global information, avoiding the multiple layer route to increasing receptive fields in standard architectures, e.g. in (Romero et al., 2014; Srivastava et al., 2015).
As expected, networks can always improve classification performance by adding more CNN modules (going from architecture A to D). Network D with 19 convolutional layers performs better than the ResNet-110 (by 0.3% top1 error), (though Network D has more parameters than the ResNet110) and is slightly worse than ResNet-164 (by 0.25% top1 error). Thus, following this trend, it is reasonable to expect a benefit if L-RNN Modules are combined with very deep networks, like the residual variants.
Networks with L-RNN modules interleaved with CNN modules. Comparing the performance of Baseline-CNN-Max (8.48%) with that of Network E (5.96%), there is a significant performance boost (2.5%), brought by simply inserting L-RNN modules. Network E also has other advantages over the networks A to D: the number of parameters, network depth, and running time. Furthermore, when we continue increasing the network depth and interleaving L-RNN modules, Network F achieves comparable results (5.39%) to ResNet-164 (5.46%) and with fewer parameters (1.55M vs. 1.7M ). This confirms that, firstly, L-RNN modules can be combined with very deep networks; and secondly, rather than hand-craft the kernel size, we should set the model free and learn contextual information at any stage.
5.2 SEMANTIC SEGMENTATION
In this section, we insert L-RNN modules into the VGG-16 networks (pre-trained on ImageNet (Deng et al., 2009)), and fine-tune the entire network for the PASCAL VOC 2012 segmentation task. The objective is to boost the segmentation performance by providing contextual information via the L-RNNs. In particular, we consider the two FCN segmentation architectures originally introduced by Long et al. (2015), FCN-32s and FCN-8s; these are described below.
We proceed in three steps: first, we establish baselines by training our own FCN-32s and FCN-8s (Appendix C), and comparing their performance to those of (Long et al., 2015). We also investigate the loss in performance as the fully connected (FC) layer is gradually reduced from 4096 to 512 channels. The reason for doing this is that when we insert the L-RNN module, its complexity (dimension of the hidden units) depends on this number of channels, and so the overall complexity can be varied. In the second step, we insert L-RNNs into the FCN-32s architecture and evaluate the change in performance. Finally, we insert L-RNNs into the FCN-8s architecture and compare with previous published methods.
Dataset & Evaluation. We used a training set consisted of VOC2012 training data (1464 images provided by the challenge organizers), and augmented with training and validation data from Hariharan et al. (2014), which further extend the training set to a total of 11, 685 images with pixellevel annotation. After removing the overlapping images between VOC2012 validation data and this dataset, we are left with 346 images from the original VOC2012 validation set to validate our model. In all the following experiments, we use a single scale for the input images (384 384), and only horizontal flipping is used for data augmentation. The performance is measured in terms of pixel intersection-over-union (IOU) averaged across the 21 classes.
5.2.1 BASELINE ARCHITECTURES AND TRAINING
Architecture & Training. In the FCN-32s, input images are passed through the whole networks, and end up with predictions of 12 12 21, then, up-sampling layers are directly used to map the predictions back to 384 384 (32 times). In the FCN-16s, instead of directly up-sampling 32 times, the predictions are first up-sampled by 2, and summed up with stream predictions from pool4 (named after VGG16), then up-sampled by 16 times. In the FCN-8s, the stream predictions from pool3 are further added to the results from FCN-16s, thus, up-sampling layers with only factor 8 is needed. (Appendix C)
For all the architectures, the base net(VGG16) is pre-trained on ImageNet (Deng et al., 2009), we further train on Pascal VOC2012 for 50 epochs, similar to the experiment for CIFAR-10, we iteratively increase or decrease the learning rate between 10 3 and 10 5 after every 10 epochs. The 4096 channel architectures are trained first, and then the number of channels is gradually reduced in the FC layer by randomly cutting them (e.g. from 4096 to 2048), and re-training the networks.
Results & Discussion. Table 3 shows the performance of the six baselines: FCN-32s and FCN8s with the number of channels varying from 512 to 4096. We observe that reducing the nodes in the FC layers does produce a performance drop (from 4096 to 1024 nodes, 1% mean IOU) in both FCN-32s and FCN-8s. Although from 1024 to 4096 nodes, the improvement is tiny, the difference in the number of parameters is over 64 million. Consequently, in the following experiments we choose to perform experiments based on networks with 512, 1024 or 2048 channels only (i.e. not 4096). In comparison to the original performance for the FCN-8s architecture in (Long et al., 2015), we exceed this (by 64.4 to 61.3 mean IOU) in our training. Thus, we use our trained networks as a baseline.
5.2.2 FCN-32S WITH L-RNN MODULES
Architecture & Training. The architecture FCN-32s(L-RNN) is shown in figure 4, the convolutional part of the architecture is initialized with the pre-trained FCN-32s(2048 channels in FC layer) baseline. Then, two 1D spatial RNNs are inserted into the fc1 layer in the horizontal direction, and two 1D spatial RNNs are inserted into the fc2 layer in the vertical direction. The convolution activations of fc1 are shared for both left-right and right-left scanning. Similarly for fc2, the convolution activations are shared for up-down and down-up scanning. Thus the fc1 and fc2 layers together with the added 1D spatial RNNs form a complete L-RNN module.
During training, as described in section 4, the 1D spatial RNNs are initialized with a zero recurrence matrix. The entire network is then fine-tuned end-to-end with the PASCAL VOC2012 data. We adopt RMS-prop (Tieleman & Hinton, 2012) for 30 epochs with hyper-parameters lr 10 4, ρ 0.9, 10 8, then decrease the learning rate to lr 10 5 for 10 epochs.
Results & Discussion. The results are shown in Table 3. Compare the 32s rows with and without the L-RNN for the FC layers with 512, 1024, and 2048 channels. As can be seen, the addition of the L-RNN always improve the segmentation performance over the pre-trained FCN-32s baselines. However, the improvement is not large – about 1 1.5% mean IOU. This is because the receptive field in the fully connected layers of FCN-32s is sufficiently large to cover 224 224 pixels of the input patch, and consequenly the networks are not able to benefit much from the context provided by the L-RNN. The benefit is greater when L-RNNs are added to the lower layers (where the receptive fields of the convolutions is much smaller), and we turn to that case next.
5.2.3 FCN-8S WITH L-RNN MODULES
Architecture & Training. The architecture FCN-8s(L-RNN) is shown in figure 4, as with the FCN-32s architecture, 1D spatial RNNs are inserted into the fc1 and fc2 layers to form a L-RNN module. L-RNNs are also inserted into the lower layers, namely pool3 and pool4 layers. Unlike the FC layers in the FCN-32s, where prediction for each central pixel comes from image patches of size 224 224, the predictions from pool3 and pool4 are based on receptive field on the image of much smaller sizes (around 44 44 and 100 100 pixels respectively). Thus, the inserted L-RNN modules must be able to model relatively long-range dependencies.
During training, the network is initialized from the FCN-8s baseline, and then fine-tuned using segmentation data. Again the PASCAL VOC dataset is used. Furthermore, when comparing to the other previously published methods, the network is further trained on the COCO trainval dataset, and we use a densely connected CRF as post-processing (Krhenbhl & Koltun, 2012).
Results on PASCAL VOC Validation set. The experimental results are shown in Table 3.
Comparing the rows for 32s with and without L-RNN, to those for 8s with and without L-RNN. We can draw the following conclusions:
Improvement due to the skip layers. It can be seen (for IOU) that going from FCN-32s(2048) to FCN-8s(2048), where there are additional skip layers, the performance is boosted from 62.7 to 64.1. The skip layers in the FCN-8s architecture introduce more parameters, but this is not the reason for the performance boost since FCN-8s(2048) and FCN-32s(4096), have a similar number of parameters though they perform very differently (64.1 vs. 62.9). This observation confirms that the performance gain is brought by the the skip layers, rather than the increased number of parameters.
Improvement due to L-RNN module. Inserting a L-RNN to the FC layers of FCN-32s(2048), only improves the performance from 62.7 to 64.2. However, as noted earlier, since the nodes in the
FC layers already cover the entire input patch of size 224 224, the L-RNN can contribute only little context here.
In contrast, adding L-RNNs to FCN-8s brings a substantial improvement from 64.1(FCN-8s) to 69.1(FCN-8s-LRNN). This process will introduce more parameters due to the recurrence term in the RNNs, but it is clear that the improvement is mainly from the inserted L-RNN module after pool3 and pool4 in FCN-8s, rather than from the increased number of parameters. The reason is that, when comparing FCN-8s (2048 channels without L-RNN) to FCN-8s (4096 channels without L-RNN), although the number of parameters is increased dramatically, the performance is only increased from 64.1 to 64.4. While FCN-8s (4096 channels without L-RNN) has roughly the same number of parameters as that of FCN-8s (2048 channels with L-RNN), but the performance gain is from 64.4 to 69.1. In conclusion, the L-RNN is able to learn contextual information over a much larger range than the receptive field of pure local convolutions.
Results on PASCAL VOC Test set. Table 4 shows the results of the FCN-8s with L-RNNs on the PASCAL VOC test data, and also compares to others who have published on this dataset. The performance is far superior to the original result (Long et al., 2015) using a FCN-8s with 4096 channels (whereas only 2048 channels are used here). We also compare to the dilated convolution network of (Yu & Koltun, 2016), obtaining comparable, though slightly better performance. Note that in (Yu & Koltun, 2016), multi-scale contextual information is captured by explicitly designing dilated convolution kernels, while the L-RNN is able to learn contextual information implicitly. Finally, we compare to (Zheng et al., 2015) who add a densely connected CRF to FCN-8s. If we also add a dense CRF as post-processing, we boost the performance by 1% in IOU (the same boost as obtained by (Yu & Koltun, 2016)). In Figure 5, we show the samples of semantic segmentations on
the PASCAL VOC2012 validation set. In each figure, we show our predictions and the results after CRF post-processing. Comparing with the end-to-end trainable CRF-RNN (Zheng et al., 2015), our predictions miss the small details, like the wheel of the bicycle, but show much better performance in determining the class of the segmented regions – something that context can really contribute to.
6 CONCLUSION & FUTURE WORK
This paper has shown that the proposed L-RNN module is an alternative way of adding multi-level spatial context to a network. In fact, L-RNNs can be interleaved with convolutional layers to learn context at any stage. When the L-RNN is only used at the final stage after the CNNs, it gives shallow networks the receptive fields of far deeper networks. Furthermore, we have demonstrated that inserting L-RNNs can boost the performance of pre-trained networks, and given an initialization procedure that makes this training a simple matter of end-to-end fine tuning.
There is much left to investigate using L-RNNs as a new building block, and we suggest some avenues here: (i) training the hybrid architectures on larger dataset, such as ImageNet (Deng et al., 2009), and learn representations that can be transferred to other vision tasks, (ii) a similar investigation for deep residual networks where the residual blocks are either convolutional or L-RNNs; and (iii) including a CRF final layer in end-to-end training.
A TRAINING DETAILS FOR CIFAR-10
A.1 RNN WITH GATED RECURRENT UNITS
In the Layer-RNN, we test the gated recurrent units (GRU) for the RNN blocks (Chung et al., 2015), the GRU has two gates, namely reset gate r and update gate z. Intuitively, the reset gate determines how to combine the new input with the previous memory, and the update gate defines how much of the previous memory to use, thus, the hidden state st of the GRU at time t can be computed as :
z σpxtU z st 1W zq (10) r σpxtU r st 1W rq (11)
h fpxtU h pst 1 rqW hq (12) st p1 zq h z st 1 (13)
A.2 VANILLA RNN WITH LAYER NORMALIZATION
To simplify the training process and reduce number of parameters, we also test the vanilla RNNs for the RNN blocks with Layer Normalization(Ba et al., 2016). In a standard RNN, the outputs in the recurrent layer are calculated from the current input xt and the previous hidden states ht 1, which are denoted as at Uxt V ht 1. The layer normalized layer is computed as :
ht fp g
σt pat µtq bq (14)
µt 1
H
Ḩ
i 1
ait σt gffe 1 H Ḩ
i 1
pait µtq 2 (15)
Where U is the current input-to-hidden term, and V is the hidden-to-hidden recurrence term, b and g are defined as the bias and gain parameters of the same dimension as ht.
During training, we iteratively increase and decrease the learning rate (learning rate restart) between 10 3 and 10 5 based on the conjecture that (Figure 6), networks tend to get trapped in the regions with small derivatives, such as saddle points or bad local minima Dauphin et al. (2014). Traditionally, the learning rate is decreased every several epochs, and gradients that are used to update parameters depend on both the learning rate and the derivatives w.r.t loss functions. At the end of training, both of these two terms tend to be very small. Therefore, it becomes difficult for the networks to escape from these regions. During our training, we restart the learning rate every some epochs (we try 60 or 80 in our training), and decrease it gradually.
Deep Neural Networks may easily be trapped into saddle point or bad local minima.
B FINE-TUNING LAYER-RNNS WITH ZERO RECURRENCE MATRIX
In this section, we derive the procedure for fine-tuning the recurrence matrix, when it is initialized as zeros. We will only consider 1D scan-lines of the spatial RNN, and therefore simplify the derivation to a 1D sequence. Consider the fully connected layer for simplicity, L,L 1 denote layer, t refers to the index of input, f refers to ReLU, U, V refer to the input-hidden matrix and recurrence matrix respectively.
st UX L t V X L 1 t 1 b (16)
XL 1t fpstq (17)
AssumeE denotes the loss function for a specific task. Since V is shared for the whole 1D sequence (length denoted by T), the back-propagation within the layer L 1 can then be derived as:
BE BV ¸ T ¸ t¤T BE BXL 1T BXL 1T BXL 1t BXL 1t Bst Bst BV
(18)
where,
BXL 1T BXL 1t
BXL 1T BXL 1T 1 BXL 1T 1 BXL 1T 2 . . . BXL 1t 1 BXL 1t and BXL 1t 1 BXL 1t V T diagpf 1q (19)
Each Jacobian BXL 1t 1
BXL 1t is a product of two matrices: (a) the recurrence weight matrix V , and (b) the
diagonal matrix composed of the derivative of ReLU (f 1). Therefore, when V is initialized to zero at the starting point, there are no long-range dependencies, and for t   T , BXL 1t 1
BXL 1t will be zero, while
for t T :
BE BV ¸ T BE BXL 1T BXL 1T BsT BsT BV where BXL 1T BsT f 1, BsT BV
XL 1T 1 (20)
V1 V0 α BE
BV gradient descent at first iteration (21)
Since V0 is initialized as zero, V1 α BEBV . In other words, instead of initializing the recurrence matrix V randomly or to be identity matrix, we actually initialize it based on the features in a local neighbourhood (equation 20). During the back-propagation of spatial RNNs, gradients flow within layers, BE
BU (between layers) is calculated in the same way as normal convolutional layers.
C DETAILED DESCRIPTION OF THE FCNS USED IN THE PAPER
The complete FCNs architecture used in the paper.

The paper proposes a method of integrating recurrent layers within larger, potentially pre-trained, convolutional networks. The objective is to combine the feature extraction abilities of CNNs with the ability of RNNs to gather global context information.
The authors validate their idea on two tasks, image classification (on CIFAR-10) and semantic segmentation (on PASCAL VOC12).
On the positive side, the paper is clear and well-written (apart from some occasional typos), the proposed idea is simple and could be adopted by other works, and can be deployed as a beneficial perturbation of existing systems, which is practically important if one wants to increase the performance of a system without retraining it from scratch. The evaluation is also systematic, providing a clear ablation study.
On the negative side, the novelty of the work is relatively limited, while the validation is lacking a bit.
Regarding novelty, the idea of combining a recurrent layer with a CNN, something practically very similar was proposed in Bell et al (2016). There are a few technical differences (e.g. cascading versus applying in parallel the recurrent layers), but in my understanding these are minor changes. The idea of initializing the recurrent network with the CNN is reasonable but is at the level of improving one wrong choice in the original work of Bell, rather than really proposing something novel.
This contribution (" we use RNNs within layers") is repeatedly mentioned in the paper (including intro &  conclusion), but in my understanding was part of Bell et al, modulo minor changes.
Regarding the evaluation, experiments on CIFAR are interesting, but only as proof of concept.
Furthermore, as noted in my early question, Wide Residual Networks (Sergey Zagoruyko, Nikos Komodakis, BMVC16)
report  better results on CIFAR-10 (4% error), while not using any recurrent layers (rather using instead a wide, VGG-type, ResNet variant). So.
The authors answer: "Wide Residual Networks use the depth of the network to spread the receptive field across the entire image (DenseNet (Huang et al., 2016) similarly uses depth). Thus there is no need for recurrence within layers to capture contextual information. In contrast, we show that a shallow CNN, where the receptive field would be limited, can capture contextual information within the whole image if a L-RNN is used."
So, we agree that WRN do not need recurrence - and can still do better.
The point of my question has practically been whether using a recurrent layer is really necessary; I can understand the answer as being "yes, if you want to keep your network shallow".  I do not necessarily see why one would want to keep one's network shallow.
Probably an evaluation on imagenet would bring some more insight about the merit of this layer.
Regarding semantic segmentation, one of my questions has been:
"Is the boost you are obtaining due to something special to the recurrent layer, or is simply because one is adding extra parameters on top of a pre-trained network? (I admit I may have missed some details of your experimental evaluation)"
The answer was:
"...For PASCAL segmentation, we add the L-RNN into a pre-trained network (this adds recurrence parameters), and again show that this boosts performance - more so than adding the same number of parameters as extra CNN layers - as it is able to model long-range dependences"
I could not find one such experiment in the paper ('more so than adding the same number of parameters as extra CNN layers'); I understand that you have 2048 x 2048 connections for the recurrence, it would be interesting to see what you get by spreading them over (non-recurrent) residual layers.
Clearly, this is not going to be my criterion for rejection/acceptance, since one can easily make it fail - but I was mostly asking for some sanity check
Furthermore, it is a bit misleading to put in Table 3 FCN-8s and FCN8s-LRNN, since this gives the impression that the LRNN gives a  boost by 10%. In practice the "FCN8s" prefix of "FCN8s-LRNN" is that of the authors, and not of Long et al (as indicated in Table 2, 8s original is quite worse than 8s here).
Another thing that is not clear to me is where the boost comes from in Table 2; the authors mention that "when inserting the L-RNN after pool 3 and pool4 in FCN-8s, the L-RNN is able to learn contextual information over a much larger range than the receptive field of pure local convolutions. "
This is potentially true, but I do not see why this was not also the case for FCN-32s (this is more a property of the recurrence rather than the 8/32 factor, right?)
A few additional points:
It seems like Fig 2b and Fig2c never made it into the pdf.
Figure 4 is unstructured and throws some 30 boxes to the reader - I would be surprised if anyone is able to get some information out of this (why not have a table?)
Appendix A: this is very mysterious. Did you try other learning rate schedules? (e.g. polynomial)
What is the performance if you apply a standard training schedule? (e.g. step).
Appendix C: "maps .. is" -> "maps ... are"