Yes, they do. This paper provides the first empirical demonstration that deep convolutional models really need to be both deep and convolutional, even when trained with methods such as distillation that allow small or shallow models of high accuracy to be trained. Although previous research showed that shallow feed-forward nets sometimes can learn the complex functions previously learned by deep nets while using the same number of parameters as the deep models they mimic, in this paper we demonstrate that the same methods cannot be used to train accurate models on CIFAR-10 unless the student models contain multiple layers of convolution. Although the student models do not have to be as deep as the teacher model they mimic, the students need multiple convolutional layers to learn functions of comparable accuracy as the deep convolutional teacher.
1 INTRODUCTION
Cybenko (1989) proved that a network with a large enough single hidden layer of sigmoid units can approximate any decision boundary. Empirical work, however, suggests that it can be difficult to train shallow nets to be as accurate as deep nets. Dauphin and Bengio (2013) trained shallow nets on SIFT features to classify a large-scale ImageNet dataset and found that it was difficult to train large, high-accuracy, shallow nets. A study of deep convolutional nets suggests that for vision tasks deeper models are preferred under a parameter budget (e.g. Eigen et al. (2014); He et al. (2015); Simonyan and Zisserman (2014); Srivastava et al. (2015)). Similarly, Seide et al. (2011) and Geras et al. (2015) show that deeper models are more accurate than shallow models in speech acoustic modeling. More recently, Romero et al. (2015) showed that it is possible to gain increases in accuracy in models with few parameters by training deeper, thinner nets (FitNets) to mimic much wider nets. Cohen and Shashua (2016); Liang and Srikant (2016) suggest that the representational efficiency of deep networks scales exponentially with depth, but it is unclear if this applies only to pathological problems, or is encountered in practice on data sets such as TIMIT and CIFAR.
Ba and Caruana (2014), however, demonstrated that shallow nets sometimes can learn the functions learned by deep nets, even when restricted to the same number of parameters as the deep nets. They did this by first training state-of-the-art deep models, and then training shallow models to mimic the deep models. Surprisingly, and for reasons that are not well understood, the shallow models learned more accurate functions when trained to mimic the deep models than when trained on the original data used to train the deep models. In some cases shallow models trained this way were as accurate as state-of-the-art deep models. But this demonstration was made on the TIMIT speech recognition benchmark. Although their deep teacher models used a convolutional layer, convolution is less important for TIMIT than it is for other domains such as image classification.
Ba and Caruana (2014) also presented results on CIFAR-10 which showed that a shallow model could learn functions almost as accurate as deep convolutional nets. Unfortunately, the results on CIFAR-10 are less convincing than those for TIMIT. To train accurate shallow models on CIFAR-10
they had to include at least one convolutional layer in the shallow model, and increased the number of parameters in the shallow model until it was 30 times larger than the deep teacher model. Despite this, the shallow convolutional student model was several points less accurate than a teacher model that was itself several points less accurate than state-of-the-art models on CIFAR-10.
In this paper we show that the methods Ba and Caruana used to train shallow students to mimic deep teacher models on TIMIT do not work as well on problems such as CIFAR-10 where multiple layers of convolution are required to train accurate teacher models. If the student models have a similar number of parameters as the deep teacher models, high accuracy can not be achieved without multiple layers of convolution even when the student models are trained via distillation.
To ensure that the shallow student models are trained as accurately as possible, we use Bayesian optimization to thoroughly explore the space of architectures and learning hyperparameters. Although this combination of distillation and hyperparameter optimization allows us to train the most accurate shallow models ever trained on CIFAR-10, the shallow models still are not as accurate as deep models. Our results clearly suggest that deep convolutional nets do, in fact, need to be both deep and convolutional, even when trained to mimic very accurate models via distillation (Hinton et al., 2015).
2 TRAINING SHALLOW NETS TO MIMIC DEEPER CONVOLUTIONAL NETS
In this paper, we revisit the CIFAR-10 experiments in Ba and Caruana (2014). Unlike in that work, here we compare shallow models to state-of-the-art deep convolutional models, and restrict the number of parameters in the shallow student models to be comparable to the number of parameters in the deep convolutional teacher models. Because we anticipated that our results might be different, we follow their approach closely to eliminate the possibility that the results differ merely because of changes in methodology. Note that the goal of this paper is not to train models that are small or fast as in Bucila et al. (2006), Hinton et al. (2015), and Romero et al. (2015), but to examine if shallow models can be as accurate as deep convolutional models given the same parameter budget.
There are many steps required to train shallow student models to be as accurate as possible: train state-of-the-art deep convolutional teacher models, form an ensemble of the best deep models, collect and combine their predictions on a large transfer set, and then train carefully optimized shallow student models to mimic the teacher ensemble. For negative results to be informative, it is important that each of these steps be performed as well as possible. In this section we describe the experimental methodology in detail. Readers familiar with distillation (model compression), training deep models on CIFAR-10, data augmentation, and Bayesian hyperparameter optimization may wish to skip to the empirical results in Section 3.
2.1 MODEL COMPRESSION AND DISTILLATION
The key idea behind model compression is to train a compact model to approximate the function learned by another larger, more complex model. Bucila et al. (2006) showed how a single neural net of modest size could be trained to mimic a much larger ensemble. Although the small neural nets contained 1000× fewer parameters, often they were as accurate as the large ensembles they were trained to mimic.
Model compression works by passing unlabeled data through the large, accurate teacher model to collect the real-valued scores it predicts, and then training a student model to mimic these scores. Hinton et al. (2015) generalized the methods of Bucila et al. (2006) and Ba and Caruana (2014) by incorporating a parameter to control the relative importance of the soft targets provided by the teacher model to the hard targets in the original training data, as well as a temperature parameter that regularizes learning by pushing targets towards the uniform distribution. Hinton et al. (2015) also demonstrated that much of the knowledge passed from the teacher to the student is conveyed as dark knowledge contained in the relative scores (probabilities) of outputs corresponding to other classes, as opposed to the scores given to just the output for the one correct class.
Surprisingly, distillation often allows smaller and/or shallower models to be trained that are nearly as accurate as the larger, deeper models they are trained to mimic, yet these same small models are not as accurate when trained on the 1-hot hard targets in the original training set. The reason for this is not yet well understood. Similar compression and distillation methods have also successfully
been used in speech recognition (e.g. Chan et al. (2015); Geras et al. (2015); Li et al. (2014)) and reinforcement learning Parisotto et al. (2016); Rusu et al. (2016). Romero et al. (2015) showed that distillation methods can be used to train small students that are more accurate than the teacher models by making the student models deeper, but thinner, than the teacher model.
2.2 MIMIC LEARNING VIA L2 REGRESSION ON LOGITS
We train shallow mimic nets using data labeled by an ensemble of deep teacher nets trained on the original 1-hot CIFAR-10 training data. The deep teacher models are trained in the usual way using softmax outputs and cross-entropy cost function. Following Ba and Caruana (2014), the student mimic models are not trained with cross-entropy on the ten p values where pk = ezk/ ∑ j e
zj output by the softmax layer from the deep teacher model, but instead are trained on the un-normalized log probability values z (the logits) before the softmax activation. Training on the logarithms of predicted probabilities (logits) helps provide the dark knowledge that regularizes students by placing emphasis on the relationships learned by the teacher model across all of the outputs.
As in Ba and Caruana (2014), the student is trained as a regression problem given training data {(x(1), z(1)),...,(x(T ), z(T ))}:
L(W ) = 1 T ∑ t ||g(x(t);W )− z(t)||22, (1)
whereW represents all of the weights in the network, and g(x(t);W ) is the model prediction on the tth training data sample.
2.3 USING A LINEAR BOTTLENECK TO SPEED UP TRAINING
A shallow net has to have more hidden units in each layer to match the number of parameters in a deep net. Ba and Caruana (2014) found that training these wide, shallow mimic models with backpropagation was slow, and introduced a linear bottleneck layer between the input and non-linear layers to speed learning. The bottleneck layer speeds learning by reducing the number of parameters that must be learned, but does not make the model deeper because the linear terms can be absorbed back into the non-linear weight matrix after learning. See Ba and Caruana (2014) for details. To match their experiments we use linear bottlenecks when training student models with 0 or 1 convolutional layers, but did not find the linear bottlenecks necessary when training student models with more than 1 convolutional layer.
2.4 BAYESIAN HYPERPARAMETER OPTIMIZATION
The goal of this work is to determine empirically if shallow nets can be trained to be as accurate as deep convolutional models using a similar number of parameters in the deep and shallow models. If we succeed in training a shallow model to be as accurate as a deep convolutional model, this provides an existence proof that shallow models can represent and learn the complex functions learned by deep convolutional models. If, however, we are unable to train shallow models to be as accurate as deep convolutional nets, we might fail only because we did not train the shallow nets well enough.
In all our experiments we employ Bayesian hyperparameter optimization using Gaussian process regression to ensure that we thoroughly and objectively explore the hyperparameters that govern learning. The implementation we use is Spearmint (Snoek et al., 2012). The hyperparameters we optimize with Bayesian optimization include the initial learning rate, momentum, scaling of the initial random weights, scaling of the inputs, and terms that determine the width of each of the network’s layers (i.e. number of convolutional filters and neurons). More details of the hyperparameter optimization can be found in Sections 2.5, 2.7, 2.8 and in the Appendix.
2.5 TRAINING DATA AND DATA AUGMENTATION
The CIFAR-10 (Krizhevsky, 2009) data set consists of a set of natural images from 10 different object classes: airplane, automobile, bird, cat, deer, dog, frog, horse, ship, truck. The dataset is a labeled subset of the 80 million tiny images dataset (Torralba et al., 2008) and is divided into 50,000 train and
10,000 test images. Each image is 32×32 pixels in 3 color channels, yielding input vectors with 3072 dimensions. We prepared the data by subtracting the mean and dividing by the standard deviation of each image vector. We train all models on a subset of 40,000 images and use the remaining 10,000 images as the validation set for the Bayesian optimization. The final trained models only used 80% of the theoretically available training data (as opposed to retraining on all of the data after hyperparameter optimization).
We employ the HSV-data augmentation technique as described by Snoek et al. (2015). Thus we shift hue, saturation and value by uniform random values: ∆h ∼ U(−Dh, Dh), ∆s ∼ U(−Ds, Ds), ∆v ∼ U(−Dv, Dv). Saturation and value values are scaled globally: as ∼ U( 11+As , 1 + As), av ∼ U( 1 1+Av
, 1 + Av). The five constants Dh, Ds, Dv, As, Av are treated as additional hyperparameters in the Bayesian hyperparameter optimization.
All training images are mirrored left-right randomly with a probability of 0.5. The input images are further scaled and jittered randomly by cropping windows of size 24×24 up to 32×32 at random locations and then scaling them back to 32×32. The procedure is as follows: we sample an integer value S ∼ U(24, 32) and then a pair of integers x, y ∼ U(0, 32 − S). The transformed resulting image is R = fspline,3(I[x : x + S, y : y + S]) with I denoting the original image and fspline,3 denoting the 3rd order spline interpolation function that maps the 2D array back to 32×32 (applied to the three color channels separately).
All data augmentations for the teacher models are computed on the fly using different random seeds. For student models trained to mimic the ensemble (see Section 2.7 for details of the ensemble teacher model), we pre-generated 160 epochs worth of randomly augmented training data, evaluated the ensemble’s predictions (logits) on these samples, and saved all data and predictions to disk. All student models thus see the same training data in the same order. The parameters for HSV-augmentation in this case had to be selected beforehand; we chose to use the settings found with the best single model (Dh = 0.06, Ds = 0.26, Dv = 0.20, As = 0.21, Av = 0.13). Pre-saving the logits and augmented data is important to reduce the computational cost at training time, and to ensure that all student models see the same training data
Because augmentation allows us to generate large training sets from the original 50,000 images, we use augmented data as the transfer set for model compression. No extra unlabeled data is required.
2.6 LEARNING-RATE SCHEDULE
We train all models using SGD with Nesterov momentum. The initial learning rate and momentum are chosen by Bayesian optimization. The learning rate is reduced according to the evolution of the model’s validation error: it is halved if the validation error does not drop for ten epochs in a row. It is not reduced within the next eight epochs following a reduction step. Training ends if the error did not drop for 30 epochs in a row or if the learning rate was reduced by a factor of more than 2000 in total.
This schedule provides a way to train the highly varying models in a fair manner (it is not feasible to optimize all of the parameters that define the learning schedule). It also decreases the time spent to train each model compared to using a hand-selected overestimate of the number of epochs to train, thus allowing us to train more models in the hyperparameter search.
2.7 SUPER TEACHER: AN ENSEMBLE OF 16 DEEP CONVOLUTIONAL CIFAR-10 MODELS
One limitation of the CIFAR-10 experiments performed in Ba and Caruana (2014) is that the teacher models were not state-of-the-art. The best deep models they trained on CIFAR-10 had only 88% accuracy, and the ensemble of deep models they used as a teacher had only 89% accuracy. The accuracies were not state-of-the-art because they did not use augmentation and because their deepest models had only three convolutional layers. Because our goal is to determine if shallow models can be as accurate as deep convolutional models, it is important that the deep models we compare to (and use as teachers) are as accurate as possible.
We train deep neural networks with eight convolutional layers, three intermittent max-pooling layers and two fully-connected hidden layers. We include the size of these layers in the hyperparameter optimization, by allowing the first two convolutional layers to contain from 32 to 96 filters each, the next two layers to contain from 64 to 192 filters, and the last four convolutional layers to contain
from 128 to 384 filters. The two fully-connected hidden layers can contain from 512 to 1536 neurons. We parametrize these model-sizes by four scalars (the layers are grouped as 2-2-4) and include the scalars in the hyperparameter optimization. All models are trained using Theano (Bastien et al., 2012; Bergstra et al., 2010).
We optimize eighteen hyperparameters overall: initial learning rate on [0.01, 0.05], momentum on [0.80, 0.91], l2 weight decay on [5 · 10−5,4 · 10−4], initialization coefficient on [0.8, 1.35] which scales the initial weights of the CNN, four separate dropout rates, five constants controlling the HSV data augmentation, and the four scaling constants controlling the networks’ layer widths. The learning rate and momentum are optimized on a log-scale (as opposed to linear scale) by optimizing the exponent with appropriate bounds, e.g. LR = e−x optimized over x on [3.0, 4.6]. See the Appendix for more details about hyperparameter optimization.
We trained 129 deep CNN models with Spearmint. The best model obtained an accuracy of 92.78%; the fifth best achieved 92.67%. See Table 1 for the sizes and architectures of the three best models.
We are able to construct a more accurate model on CIFAR-10 by forming an ensemble of multiple deep convolutional neural nets, each trained with different hyperparameters, and each seeing slightly different training data (as the augmentation parameters vary). We experimented with a number of ensembles of the many deep convnets we trained, using accuracy on the validation set to select the best combination. The final ensemble contained 16 deep convnets and had an accuracy of 94.0% on the validation set, and 93.8% on the final test set. We believe this is among the top published results for deep learning on CIFAR-10. The ensemble averages the logits predicted by each model before the softmax layers.
We used this very accurate ensemble model as the teacher model to label the data used to train the shallower student nets. As described in Section 2.2, the logits (the scores just prior to the final softmax layer) from each of the CNN teachers in the ensemble model are averaged for each class, and the average logits are used as final regression targets to train the shallower student neural nets.
2.8 TRAINING SHALLOW STUDENT MODELS TO MIMIC AN ENSEMBLE OF DEEP CONVOLUTIONAL MODELS
We trained student mimic nets with 1, 3.161, 10 and 31.6 million trainable parameters on the pre-computed augmented training data (Section 2.5) that was re-labeled by the teacher ensemble (Section 2.7). For each of the four student sizes we trained shallow fully-connected student MLPs containing 1, 2, 3, 4, or 5 layers of non-linear units (ReLU), and student CNNs with 1, 2, 3 or 4 convolutional layers. The convolutional student models also contain one fully-connected ReLU layer. Models with zero or only one convolutional layer contain an additional linear bottleneck layer to speed up learning (cf. Section 2.3). We did not need to use a bottleneck to speed up learning for the deeper models as the number of learnable parameters is naturally reduced by the max-pooling layers.
The student CNNs use max-pooling and Bayesian optimization controls the number of convolutional filters and hidden units in each layer. The hyperparameters we optimized in the student models are: initial learning rate, momentum, scaling of the initially randomly distributed learnable parameters, scaling of all pixel values of the input, and the scale factors that control the width of all hidden and convolutional layers in the model. Weights are initialized as in Glorot and Bengio (2010). We intentionally do not optimize and do not make use of weight decay and dropout when training student models because preliminary experiments showed that these consistently reduced the accuracy of student models by several percent. Please refer to the Appendix for more details on the individual architectures and hyperparameter ranges.
3 EMPIRICAL RESULTS
Table 1 summarizes results after Bayesian hyperparameter optimization for models trained on the original 0/1 hard CIFAR-10 labels. All of these models use weight decay and are trained with the dropout hyperparameters included in the Bayesian optimization. The table shows the accuracy of the best three deep convolutional models we could train on CIFAR-10, as well as the accuracy of
13.16 ≈ Sqrt(10) falls halfway between 1 and 10 on log scale.
the ensemble of 16 deep CNNs. For comparison, the accuracy of the ensemble trained by Ba and Caruana (2014)) is included at the bottom of the table.
Table 2 summarizes the results after Bayesian hyperparameter optimization for student models of different depths and number of parameters trained on soft targets (average logits) to mimic the teacher ensemble of 16 deep CNNs. For comparison, the student model trained by Ba and Caruana (2014) also is shown.
The first four rows in Table 1 show the accuracy of convolutional models with 10 million parameters and 1, 2, 3, and 4 convolutional layers. The accuracies of these same architectures with 1M, 3.16M, 10M, and 31.6M parameters when trained as students on the soft targets predicted by the teacher ensemble are shown in Table 2. Comparing the accuracies of the models with 10 million parameters in both tables, we see that training student models to mimic the ensemble leads to significantly better accuracy in every case. The gains are more pronounced for shallower models, most likely because their learnable internal representations do not naturally lead to good generalization in this task when trained on the 0/1 hard targets: the difference in accuracy for models with one convolutional layer is 2.7% (87.3% vs. 84.6%) and only 0.8% (92.6% vs. 91.8%) for models with four convolutional layers.
Figure 1 summarizes the results in Table 2 for student models of different depth, number of convolutional layers, and number of parameters when trained to mimic the ensemble teacher model. Student models trained on the ensemble logits are able to achieve accuracies previously unseen on CIFAR-10 for models with so few layers. Also, it is clear that there is a huge gap between the convolutional student models at the top of the figure, and the non-convolutional student models at the bottom of the figure: the most accurate student MLP has accuracy less than 75%, while the least accurate convolutional student model with the same number of parameters but only one convolutional layer has accuracy above 87%. And the accuracy of the convolutional student models increases further as more layers of convolution are added. Interestingly, the most accurate student MLPs with no convolutional layers have only 2 or 3 hidden layers; the student MLPs with 4 or 5 hidden layers are not as accurate.
Comparing the student MLP with only one hidden layer (bottom of the graph) to the student CNN with 1 convolutional layer clearly suggests that convolution is critical for this problem even when models are trained via distillation, and that it is very unlikely that a shallow non-convolutional model with 100 million parameters or less could ever achieve accuracy comparable to a convolutional model. It appears that if convolution is critical for teacher models trained on the original 0/1 hard targets, it
is likely to be critical for student models trained to mimic these teacher models. Adding depth to the student MLPs without adding convolution does not significantly close this “convolutional gap”.
Furthermore, comparing student CNNs with 1, 2, 3, and 4 convolutional layers, it is clear that CNN students benefit from multiple convolutional layers. Although the students do not need as many layers as teacher models trained on the original 0/1 hard targets, accuracy increases significantly as multiple convolutional layers are added to the model. For example, the best student with only one convolutional layer has 87.7% accuracy, while the student with the same number of parameters (31M) and 4 convolutional layers has 92.6% accuracy.
Figure 1 includes short horizontal lines at 10M parameters indicating the accuracy of non-student models trained on the original 0/1 hard targets instead of on the soft targets. This “compression gap” is largest for shallower models, and as expected disappears as the student models become architecturally more similar to the teacher models with multiple layers of convolution. The benefits of distillation are most significant for shallow models, yielding an increase in accuracy of 3% or more.
One pattern that is clear in the graph is that all student models benefit when the number of parameters increases from 1 million to 31 million parameters. It is interesting to note, however, that the largest student (31M) with a one convolutional layer is less accurate than the smallest student (1M) with two convolutional layers, further demonstrating the value of depth in convolutional models.
In summary, depth-constrained student models trained to mimic a high-accuracy ensemble of deep convolutional models perform better than similar models trained on the original hard targets (the “compression” gaps in Figure 1), student models need at least 3-4 convolutional layers to have high accuracy on CIFAR-10, shallow students with no convolutional layers perform poorly on CIFAR-10, and student models need at least 3-10M parameters to perform well. We are not able to compress deep convolutional models to shallow student models without significant loss of accuracy.
We are currently running a reduced set of experiments on ImageNet, though the chances of shallow models performing well on a more challenging problem such as ImageNet appear to be slim.
4 DISCUSSION
Although we are not able to train shallow models to be as accurate as deep models, the models trained via distillation are the most accurate models of their architecture ever trained on CIFAR-10. For example, the best single-layer fully-connected MLP (no convolution) we trained achieved an accuracy of 70.2%. We believe this to be the most accurate shallow MLP ever reported for CIFAR-10 (in comparison to 63.1% achieved by Le et al. (2013), 63.9% by Memisevic et al. (2015) and 64.3% by Geras and Sutton (2015)). Although this model cannot compete with convolutional models, clearly distillation helps when training models that are limited by architecture and/or number of parameters. Similarly, the student models we trained with 1, 2, 3, and 4 convolutional layers are, we believe, the most accurate convnets of those depths reported in the literature. For example, the ensemble teacher model in Ba and Caruana (2014) was an ensemble of four CNNs, each of which had 3 convolutional layers, but only achieved 89% accuracy, whereas the single student CNNs we train via distillation achieve accuracies above 90% with only 2 convolutional layers, and above 92% with 3 convolutional layers. The only other work we are aware of that achieves comparable high accuracy with non-convolutional MLPs is recent work by Lin et al. (2016). They train multi-layer Z-Lin networks, and use a powerful form of data augmentation based on deformations that we did not use.
Interestingly, we noticed that mimic networks perform consistently worse when trained using dropout. This surprised us, and suggests that training student models on the soft-targets from a teacher provides significant regularization for the student models obviating the need for extra regularization methods such as dropout. This is consistent with the observation made by Ba and Caruana (2014) that student mimic models did not seem to overfit. Hinton et al. (2015) claim that soft targets convey more information per sample than Boolean hard targets. The also suggest that the dark knowledge in the soft targets for other classes further helped regularization, and that early stopping was unnecessary. Romero et al. (2015) extend distillation by using the intermediate representations learned by the teacher as hints to guide training deep students, and teacher confidences further help regularization by providing a measure of sample simplicity to the student, akin to curriculum learning. In other work, Pereyra et al. (2017) suggest that the soft targets provided by a teacher provide a form of confidence penalty that penalizes low entropy distributions and label smoothing, both of which improve regularization by maintaining a reasonable ratio between the logits of incorrect classes.
Zhang et al. (2016) question the traditional view of regularization in deep models. Although they do not discuss distillation, they suggest that in deep learning traditional function approximation appears to be deeply intertwined with massive memorization. The multiple soft targets used to train student models have a high information density (Hinton et al., 2015) and thus provide regularization by reducing the impact of brute-force memorization.
5 CONCLUSIONS
We train shallow nets with and without convolution to mimic state-of-the-art deep convolutional nets. If one controls for the number of learnable parameters, nets containing a single fully-connected non-linear layer and no convolutional layers are not able to learn functions as accurate as deeper convolutional models. This result is consistent with those reported in Ba and Caruana (2014). However, we also find that shallow nets that contain only 1-2 convolutional layers also are unable to achieve accuracy comparable to deeper models if the same number of parameters are used in the shallow and deep models. Deep convolutional nets are significantly more accurate than shallow convolutional models, given the same parameter budget. We do, however, see evidence that model compression allows accurate models to be trained that are shallower and have fewer convolutional layers than the deep convolutional architectures needed to learn high-accuracy models from the original 1-hot hard-target training data. The question remains why extra layers are required to train accurate models from the original training data.
6 APPENDIX
6.1 DETAILS OF TRAINING THE TEACHER MODELS
Weights of trained nets are initialized as in Glorot and Bengio (2010). The models trained in Section 2.7 contain eight convolutional layers organized into three groups (2-2-4) and two fully-connected hidden layers. The Bayesian hyperparameter optimization controls four constants C1, C2, C3, H1 all in the range [0, 1] that are then linearly transformed to the number of filters/neurons in each layer. The hyperparameters for which ranges were not shown in Section 2.7 are: the four separate dropout rates (DOc1,DOc2,DOc3,DOf) and the five constants Dh, Ds, Dv, As, Av controlling the HSV data augmentation. The ranges we selected are DOc1 ∈ [0.1, 0.3],DOc2 ∈ [0.25, 0.35],DOc3 ∈ [0.3, 0.44],DOf1 ∈ [0.2, 0.65],DOf2 ∈ [0.2, 0.65], Dh ∈ [0.03, 0.11], Ds ∈ [0.2, 0.3], Dv ∈ [0.0, 0.2], As ∈ [0.2, 0.3], Av ∈ [0.03, 0.2], partly guided by Snoek et al. (2015) and visual inspection of the resulting augmentations.
The number of filters and hidden units for the models have the following bounds: 1 conv. layer: 50 - 500 filters, 200 - 2000 hidden units, number of units in bottleneck is the dependent variable. 2 conv. layers: 50 - 500 filters, 100 - 400 filters, number of hidden units is the dependent variable. 3 conv. layers: 50 - 500 filters (layer 1), 100 - 300 filters (layers 2-3), # of hidden units is dependent the variable. 4 conv. layers: 50 - 300 filters (layers 1-2), 100 - 300 filters (layers 3-4), # of hidden units is the dependent variable.
All convolutional filters in the model are sized 3×3, max-pooling is applied over windows of 2×2 and we use ReLU units throughout all our models. We apply dropout after each max-pooling layer with the three rates DOc1,DOc2,DOc3 and after each of the two fully-connected layers with the same rate DOf .
6.2 DETAILS OF TRAINING MODELS OF VARIOUS DEPTHS ON CIFAR-10 HARD 0/1 LABELS
Models in the first four rows in Table 1 are trained similarly to those in Section 6.1, and are architecturally equivalent to the four convolutional student models shown in Table 2 with 10 million parameters. The following hyperparameters are optimized: initial learning rate [0.0015, 0.025] (optimized on a log scale), momentum [0.68, 0.97] (optimized on a log scale), constants C1, C2 ∈ [0, 1] that control the number of filters or neurons in different layers, and up to four different dropout rates DOc1 ∈ [0.05, 0.4],DOc2 ∈ [0.1, 0.6],DOc3 ∈ [0.1, 0.7],DOf1 ∈ [0.1, 0.7] for the different layers. Weight decay was set to 2 · 10−4 and we used the same data augmentation settings as for the student models. We use 5×5 convolutional filters, one nonlinear hidden layer in each model and each max-pooling operation is followed by dropout with a separately optimized rate. We use 2×2 max-pooling except in the model with only one convolutional layer where we apply 3×3 pooling as this seemed to boost performance and reduces the number of parameters.
6.3 DETAILS OF TRAINING STUDENT MODELS OF VARIOUS DEPTHS ON ENSEMBLE LABELS
Our student models have the same architecture as models in Section 6.2. The model without convolutional layers consists of one linear layer that acts as a bottleneck followed by a hidden layer of ReLU units. The following hyperparameters are optimized: initial learning rate [0.0013, 0.016] (optimized on a log scale), momentum [0.68, 0.97] (optimized on a log scale), input-scale ∈ [0.8, 1.25], global initialization scale (after initialization) ∈ [0.4, 2.0], layer-width constants C1, C2 ∈ [0, 1] that control the number of filters or neurons. The exact ranges for the number of filters and implicitly resulting number of hidden units was chosen for all twenty optimization experiments independently, as architectures, number of units and number of parameters strongly interact.
For the non-convolutional models we chose a slightly different hyper-parameterization. Given that all layers (in models with “two layers” or more) are nonlinear and fully connected we treat all of them similarly from the hyperparameter-optimizer’s point of view. In order to smoothly enforce the parameter budgets without rejecting any samples from the Bayesian optimizer we instead optimize the ratios of hidden units in each layer (numbers between 0 and 1), and then re-normalize and scale them to the final number of neurons in each layer to match the target parameter budget.
Figure 2 is similar to 1 but includes preliminary results from experiments for models with 100M parameters. We are also running experiments with 300M parameters. Unfortunately, Bayesian optimization on models with 100M and 300M parameters is even more expensive than for the other points in the graph.
As expected, adding capacity to the convolutional students (top of the figure) modestly increases their accuracy. Preliminary results for the MLPs however (too preliminary to include in the graph) may not show the same increase in accuracy with increasing model size. Models with two or three hidden layers may benefit from adding capacity to each layer, but we have yet to see any benefit from adding capacity to the MLPs with four or five hidden layers.

This paper describes a careful experimental study on the CIFAR-10 task that uses data augmentation and Bayesian hyperparameter optimization to train a large number of high-quality, deep convolutional network classification models from hard (0-1) targets.  An ensemble of the 16 best models is then used as a teacher model in the distillation framework, where student models are trained to match the averaged logits from the teacher ensemble.  Data augmentation and Bayesian hyperparameter optimization is also applied in the training of the student models.  Both non-convolutional (MLP) and convolutional student models of varying depths and parameter counts are trained.  Convolutional models with the same architecture and parameter count as some of the convolutional students are also trained using hard targets and cross-entropy loss.  The experimental results show that convolutional students with only one or two convolutional layers are unable to match the results of students having more convolutional layers under the constraint that the number of parameters in all students is kept constant.
Pros
+ This is a very thorough and well designed study that make use of the best existing tools to try to answer the question of whether or not deep convolutional models need both depth and convolution.
+ It builds nicely on the preliminary results in Ba & Caruana, 2014.
Cons
- It is difficult to prove a negative, as the authors admit.  That said, this study is as convincing as possible given current theory and practice in deep learning.
Section 2.2 should state that the logits are unnormalized log-probabilities (they don't include the log partition function).
The paper does not follow the ICLR citation style.  Quoting from the template:  "When the authors or the publication are included in the sentence, the citation should not be in parenthesis (as in “See Hinton et al. (2006) for more information.”). Otherwise, the citation should be in parenthesis (as in “Deep learning shows promise to make progress towards AI (Bengio & LeCun, 2007).”)."
There are a few minor issues with English usage and typos that should be cleaned up in the final manuscript.
necessary when training student models with more than 1 convolutional layers → necessary when training student models with more than 1 convolutional layer
remaining 10,000 images as validation set → remaining 10,000 images as the validation set
evaluate the ensemble’s predictions (logits) on these samples, and save all data → evaluated the ensemble’s predictions (logits) on these samples, and saved all data
more detail about hyperparamter optimization → more detail about hyperparameter optimization
We trained 129 deep CNN models with spearmint → We trained 129 deep CNN models with Spearmint
The best model obtained an accuracy of 92.78%, the fifth best achieved 92.67%. → The best model obtained an accuracy of 92.78%; the fifth best achieved 92.67%.
the sizes and architectures of three best models → the sizes and architectures of the three best models
clearly suggests that convolutional is critical →  clearly suggests that convolution is critical
similarly from the hyperparameter-opimizer’s point of view → similarly from the hyperparameter-optimizer’s point of view


NEURAL NETWORK COMPRESSION
The success of deep learning in numerous application domains created the desire to run and train them on mobile devices. This however, conflicts with their computationally, memory and energy intense nature, leading to a growing interest in compression. Recent work by Han et al. (2015a) propose a pipeline that involves retraining, pruning and quantization of neural network weights, obtaining state-of-the-art compression rates. In this paper, we show that competitive compression rates can be achieved by using a version of ”soft weight-sharing” (Nowlan & Hinton, 1992). Our method achieves both quantization and pruning in one simple (re-)training procedure. This point of view also exposes the relation between compression and the minimum description length (MDL) principle.
1 INTRODUCTION
”Bigger is better” is the ruling maxim in deep learning land. Deep neural nets with billions of parameters are no longer an exception. Networks of such size are unfortunately not practical for mobile, on-device applications which face strong limitations with respect to memory and energy consumption. Compressing neural networks could not only improve memory and energy consumption, but also lead to less network bandwidth, faster processing and better privacy. It has been shown that large networks are heavily over-parametrized and can be compressed by approximately two orders of magnitude without significant loss of accuracy. Apparently, over-parametrization is beneficial for optimization, but not necessary for accurate prediction. This observation has opened the door for a number of highly successful compression algorithms, which either train the network from scratch (Hinton et al., 2015; Iandola et al., 2016; Courbariaux & Bengio, 2016; Courbariaux et al., 2016) or apply compression post-optimization (Han et al., 2015b;a; Guo et al., 2016; Chen et al., 2015; Wen et al., 2016).
It has been long known that compression is directly related to (variational) Bayesian inference and the minimum description principle (Hinton & Van Camp, 1993). One can show that good compression can be achieved by encoding the parameters of a model using a good prior and specifying the parameters up to an uncertainty given, optimally, by the posterior distribution. An ingenious bitsback argument can then be used to get a refund for using these noisy weights. A number of papers have appeared that encode the weights of a neural network with limited precision (say 8 bits per weight), effectively cashing in on this ”bits-back” argument (Gupta et al., 2015; Courbariaux et al., 2014; Venkatesh et al., 2016). Some authors go so far of arguing that even a single bit per weight can be used without much loss of accuracy (Courbariaux et al., 2015; Courbariaux & Bengio, 2016).
In this work we follow a different but related direction, namely to learn the prior that we use to encode the parameters. In Bayesian statistics this is known as empirical Bayes. To encourage compression of the weights toK clusters, we fit a mixture of Gaussians prior model over the weights. This idea originates from the nineties, known as soft weight-sharing (Nowlan & Hinton, 1992) where it was used to regularize a neural network. Here our primary goal is network compression, but as was
shown in Hinton & Van Camp (1993) these two objectives are almost perfectly aligned. By fitting the mixture components alongside the weights, the weights tend to concentrate very tightly around a number of cluster components, while the cluster centers optimize themselves to give the network high predictive accuracy. Compression is achieved because we only need to encodeK cluster means (in full precision) in addition to the assignment of each weight to one of these J values (using log(J) bits per weight). We find that competitive compression rates can be achieved by this simple idea.
2 MDL VIEW ON VARIATIONAL LEARNING
Model compression was first discussed in the context of information theory. The minimum description length (MDL) principle identifies the best hypothesis to be the one that best compresses the data. More specifically, it minimizes the cost to describe the model (complexity cost LC) and the misfit between model and data (error cost LE) (Rissanen, 1978; 1986). It has been shown that variational learning can be reinterpreted as an MDL problem (Wallace, 1990; Hinton & Van Camp, 1993; Honkela & Valpola, 2004; Graves, 2011). In particular, given data D = { X = {xn}Nn=1,T = {tn}Nn=1 } , a set of parameters w = {wi}Ii=1 that describes the model and an approximation q(w) of the posterior p(w|D), the variational lower bound, also known as negative variational free energy, L(q(w),w) can be decomposed in terms of error and complexity losses
L(q(w),w) = −Eq(w) [ log ( p(D|w)p(w)
q(w)
)] = Eq(w) [− log p(D|w)]︸ ︷︷ ︸
LE
+ KL(q(w)||p(w))︸ ︷︷ ︸ LC (1)
where p(w) is the prior over w and p(D|w) is the model likelihood. According to Shannon’s source coding theorem, LE lower bounds the expected amount of information needed to communicate the targets T, given the receiver knows the inputs X and the model w. The functional form of the likelihood term is conditioned by the target distribution. For example, in case of regression the predictions of the model are assumed be normally distributed around the targets T.
p(D|w) = p(T|X,w) = N∏ n=1 N (tn|xn,w) (2)
where N (tn, xn,w) is a normal distribution. Another typical example is classification where the conditional distribution of targets given data is assumed to be Bernoulli distributed1. These assumptions eventually lead to the well known error functions, namely cross-entropy error and squared error for classification and regression, respectively. Before however we can communicate the data we first seek to communicate the model. Similarly to LE , LC is a lower bound for transmitting the model. More specifically, if sender and receiver agree on a prior, LC is the expected cost of communicating the parameters w. This cost is again twofold,
KL(q(w)||p(w)) = Eq(w) [− log p(w)]−H(q(w)) (3)
where H(·) denotes the entropy. In Wallace (1990) and Hinton & Van Camp (1993) it was shown that noisy encoding of the weights can be beneficial due to the bits-back argument if the uncertainty does not harm the error loss too much. The number of bits to get refunded by an uncertain weight distribution q(w) is given by its entropy. Further, it can be shown that the optimal distribution for q(w) is the Bayesian posterior distribution. While bits-back is proven to be an optimal coding scheme (Honkela & Valpola, 2004), it is often not practical in real world settings. A practical way to cash in on noisy weights (or bits-back) is to only encode a weight value up to a limited number of bits. To see this, assume a factorized variational posteriors q(w) = ∏ q(wi). Each posterior q(wi) is associated with a Dirac distribution up to machine precision, for example, a Gaussian distribution with variance σ, for small values of σ. This implies that we formally incur a very small refund per weight,
H(q(w)) = − ∫ Ω q(w) log q(w) dw = − ∫ RI N (w|0, σI) logN (w|0, σI) = [log(2πeσ2)]I . (4)
1For more detailed discussion see Bishop (2006).
Note that the more coarse the quantization of weights the more compressible the model. The bitsback scheme makes three assumptions: (i) weights are being transmitted independently, (ii) weights are independent of each other (no mutual information), and (iii) the receiver knows the prior. Han et al. (2015a) show that one can successfully exploit (i) and (ii) by using a form of arithmetic coding (Witten et al., 1987). In particular, they employ range coding schemes such as the Sparse Matrix Format (discussed in Appendix A). This is beneficial because the weight distribution has low entropy. Note that the cost of transmitting the prior should be negligible. Thus a factorized prior with different parameters for each factor is not desirable.
The main objective of this work is to find a suitable prior for optimizing the cross-entropy between a delta posterior q(w) and the prior p(w) while at the same time keeping a practical coding scheme in mind. Recall that the cross entropy is a lower bound on the average number of bits required to encode the weights of the neural network (given infinite precision). Following Nowlan & Hinton (1992) we will model the prior p(w) as a mixture of Gaussians,
p(w) = I∏ i=1 J∑ j=0 πjN (wi|µj , σ2j ). (5)
We learn the mixture parameters µj , σj , πj via maximum likelihood simultaneously with the network weights. This is equivalent to an empirical Bayes approach in Bayesian statistics. For stateof-the-art compression schemes pruning plays a major role. By enforcing an arbitrary “zero” component to have fixed µ0 = 0 location and π0 to be close to 1, a desired weight pruning rate can be enforced. In this scenario π0 may be fixed or trainable. In the latter case a Beta distribution as hyperprior might be helpful. The approach naturally encourages quantization because in order to optimize the cross-entropy the weights will cluster tightly around the cluster means, while the cluster means themselves move to some optimal location driven by LE . The effect might even be so strong that it is beneficial to have a Gamma hyper-prior on the variances of the mixture components to prevent the components from collapsing. Furthermore, note that, mixture components merge when there is not enough pressure from the error loss to keep them separated because weights are attracted by means and means are attracted by weights hence means also attract each other. In that way the network learns how many quantization intervals are necessary. We demonstrate that behaviour in Figure 3.
3 RELATED WORK
There has been a recent surge in interest in compression in the deep neural network community. Denil et al. (2013) showed that by predicting parameters of neural networks there is great redundancy in the amount of parameters being used. This suggests that pruning, originally introduced to reduce structure in neural networks and hence improve generalization, can be applied to the problem of compression and speed-up (LeCun et al., 1989). In fact, (Han et al., 2015b; Guo et al., 2016) show that neural network survive severe weight pruning (up to 99%) without significant loss of accuracy. A variational version is is proposed by Molchanov et al. (2017), the authors learn the dropout rate for each weight in the network separately. Some parameters will effectively be pruned when the dropout rate is very high. In an approach slightly orthogonal to weight pruning, (Wen et al., 2016) applied structural regularization to prune entire sets of weights from the neural network. Such extreme weight pruning can lead to entire structures being obsolete, which for the case of convolutional filters, can greatly speed up prediction. Most importantly for compression, however, is that in conjunction with Compressed Sparse Column (CSC) format, weight pruning is a highly effective way to store and transfer weights. In Appendix A we discuss CSC format in more detail.
Reducing the bit size per stored weight is another approach to model compression. For example, reducing 32 bit floats to 1 bit leads to a 32× storage improvement. Gong et al. (2014) proposed and experimented with a number of quantization approaches: binary quantization, k-means quantization, product quantization and residual quantization. Other work finds optimal fixed points (Lin et al., 2015), applies hashing (Chen et al., 2015) or minimizes the estimation error (Wu et al., 2015). Merolla et al. (2016) demonstrates that neural networks are robust against certain amounts of low precision; indeed several groups have exploited this and showed that decreasing the weight encoding precision has little to no effect on the accuracy loss (Gupta et al., 2015; Courbariaux et al., 2014; Venkatesh et al., 2016). Pushing the idea of extreme quantization, (Courbariaux et al., 2015) and Courbariaux & Bengio (2016) trained networks from scratch that use only 1bit weights with floating point gradients; to achieve competitive results, however, they require many more of these weights.
Han et al. (2015a) elaborate on combining these ideas. They introduce an multi-step algorithm that compresses CNNS up to 49×. First, weights are pruned (giving 9− 13× compression); second they quantize the weights (increasing compression to 27− 31×); and last, they apply Huffman Encoding (giving a final compression of 35−49×). The quantization step is trainable in that after each weight is assigned to a cluster centroid, the centroids get trained with respect to the original loss function. Note that this approach has several restrictions: the number of weights set to zero is fixed after the pruning step, as is the assignment of a weight to a given cluster in the second step. Our approach overcomes all these restrictions.
A final approach to compressing information is to apply low rank matrix decomposition. First introduced by (Denton et al., 2014) and Jaderberg et al. (2014), and elaborated on by using low rank filters (Ioannou et al., 2015), low rank regularization (Tai et al., 2015) or combining low rank decomposition with sparsity (Liu et al., 2015).
4 METHOD
This section presents the procedure of network compression as applied in the experiment section. A summary can be found in Algorithm 1.
4.1 GENERAL SET-UP
We retrain pre-trained neural networks with soft weight-sharing and factorized Dirac posteriors. Hence we optimize
L(w, {µj , σj , πj}Jj=0) = LE + τLC (6) = − log p(T|X,w)− τ log p(w, {µj , σj , πj}Jj=0), (7)
via gradient descent, specifically using Adam (Kingma & Ba, 2014). The KL divergence reduces to the prior because the entropy term does not depend on any trainable parameters. Note that, similar to (Nowlan & Hinton, 1992) we weigh the log-prior contribution to the gradient by a factor of τ = 0.005. In the process of retraining the weights, the variances, means, and mixing proportions of all but one component are learned. For one component, we fix µj=0 = 0 and πj=0 = 0.999. Alternatively we can train πj=0 as well but restrict it by a Beta distribution hyper-prior. Our Gaussian MM prior is initialized with 24 + 1 = 17 components. We initialize the learning rate for the weights and means, log-variances and log-mixing proportions separately. The weights should be trained with approximately the same learning rate used for pre-training. The remaining learning rates are set to 5 · 10−4. Note that this is a very sensitive parameter. The Gaussian mixtures will collapse very fast as long as the error loss does not object. However if it collapses too fast weights might be left behind, thus it is important to set the learning rate such that the mixture does collapse too soon. If the learning rate is too small the mixture will converge too slowly. Another option to keep the mixture components from collapsing is to apply an Inverse-Gamma hyperprior on the mixture variances.
4.2 INITIALIZATION OF MIXTURE MODEL COMPONENTS
In principle, we follow the method proposed by Nowlan & Hinton (1992). We distribute the means of the 16 non-fixed components evenly over the range of the pre-trained weights. The variances will be initialized such that each Gaussian has significant probability mass in its region. A good orientation for setting a good initial variance is weight decay rate the original network has been trained on. The trainable mixing proportions are initialized evenly πj = (1 − πj=0)/J . We also experimented with other approaches such as distributing the means such that each component assumes an equal amount of probability. We did not observe any significant improvement over the simpler initialization procedure.
4.3 POST-PROCESSING
After re-training we set each weight to the mean of the component that takes most responsibility for it i.e. we quantize the weights. Before quantizing, however, there might be redundant components
as explained in section 2. To eliminate those we follow Adhikari & Hollmén (2012) by computing the KL divergence between all components. For a KL divergence smaller than a threshold, we merge two components as follows
πnew = πi + πj , µnew = πiµi + πjµj πi + πj , σ2new = πiσ
2 i + πjσ 2 j
πi + πj (8)
for two components with indices i and j. Finally, for practical compression we use the storage format used in Han et al. (2015a) (see Appendix A).
Algorithm 1 Soft weight-sharing for compression, our proposed algorithm for neural network model compression. It is divided into two main steps: network re-training and post-processing. Require: τ ← set the trade-off between error and complexity loss Require: Θ← set parameters for gradient decent scheme such as learning rate or momentum Require: α, β ← set gamma hyper-prior parameter (optional)
w← initialize network weights with pre-trained network weights θ = {µj , σj , πj}Jj=1← initialize mixture parameters (see Sec. 4.2) while w, θ not converged do
w, θ ← ∇w,θLE + τLC update w and θ with the gradient decent scheme of choice end while w ← argmax
µk πkN (w|µk, σk)∑ πjN (w|µj , σj) compute final weight by setting it to the mean that takes most
responsibility (for details see Sec. 4.3)
5 MODELS
We test our compression procedure on two neural network models used in previous work we compare against in our experiments:
(a) LeNet-300-100 an MNIST model described in LeCun et al. (1998). As no pre-trained model is available, we train our own, resulting in an error rate of 1.89%.
(b) LeNet-5-Caffe a modified version of the LeNet-5 MNIST model in LeCun et al. (1998). The model specification can be downloaded from the Caffe MNIST tutorial page 2. As no pre-trained model is available, we train our own, resulting in an error rate of 0.88%.
(c) ResNets have been invented by He et al. (2015) and further developed by He et al. (2016) and Zagoruyko & Komodakis (2016). We choose a model version of the latter authors. In accordance with their notation, we choose a network with depth 16, width k = 4 and no dropout. This model has 2.7M parameters. In our experiments, we follow the authors by using only light augmentation, i.e., horizontal flips and random shifts by up to 4 pixels. Furthermore the data is normalized. The authors report error rates of 5.02% and 24.03% for CIFAR-10 and CIFAR-100 respectively. By reimplementing their model we trained models that achieve errors 6.48% and 28.23%.
6 EXPERIMENTS
6.1 INITIAL EXPERIMENT
First, we run our algorithm without any hyper-priors, an experiment on LeNet-300-100. In Figure 1 we visualise the original distribution over weights, the final distribution over weight and how each weight changed its position in the training process. After retraining, the distribution is sharply peaked around zero. Note that with our procedure the optimization process automatically determines how many weights per layer are pruned. Specifically in this experiment, 96% of the first layer (235K
2https://github.com/BVLC/caffe/blob/master/examples/mnist/lenet_train_ test.prototxt
parameter), 90% of the second (30K) and only 18% of the final layer (10K) are pruned. From observations of this and other experiments, we conclude that the amount of pruned weights depends mainly on the number of parameters in the layer rather than its position or type (convolutional or fully connected).
Evaluating the model reveals a compression rate of 64.2. The accuracy of the model does not drop significantly from 0.9811 to 0.9806. However, we do observe that the mixture components eventually collapse, i.e., the variances go to zero. This makes the prior inflexible and the optimization can easily get stuck because the prior is accumulating probability mass around the mixture means. For a weight, escaping from those high probability plateaus is impossible. This motivates the use hyper-priors such as an Inverse-Gamma prior on the variances to essentially lower bound them.
6.2 HYPER-PARAMETER TUNING USING BAYESIAN OPTIMIZATION
The proposed procedure offers various freedoms: there are many hyper-parameters to optimize, one may use hyper-priors as motivated in the previous section or even go as far as using other distributions as mixture components.
To cope with the variety of choices, we optimize 13 hyper-parameters using the Bayesian optimization tool Spearmint Snoek et al. (2012). These include the learning rates of the weight and mixing components, the number of components, and τ . Furthermore, we assume an Inverse-Gamma prior over the variances separately for the zero component and the other components and a Beta prior over the zero mixing components.
In these experiments, we optimize re-training hyperparameters for LeNet-300-100 and LeNet-5Caffe. Due to computational restrictions, we set the number of training epochs to 40 (previously 100), knowing that this may lead to solutions that have not fully converged. Spearmint acts on an objective that balances accuracy loss vs compression rate. The accuracy loss in this case is measured over the training data. The results are shown in Figure 2. In the illustration we use the accuracy loss as given by the test data. The best results predicted by our spearmint objective are colored in dark blue. Note that we achieve competitive results in this experiment despite the restricted optimization time of 40 epochs, i.e. 18K updates.
The conclusions from this experiment are a bit unclear, on the one hand we do achieve state-ofthe-art results for LeNet-5-Caffe, on the other hand there seems to be little connection between the parameter settings of best results. One wonders if a 13 dimensional parameter space can be searched efficiently with the amount of runs we were conducting. It may be more reasonable to get more inside in the optimization process and tune parameters according to those.
6.3 COMPRESSION RESULTS
We compare our compression scheme with Han et al. (2015a) and Guo et al. (2016) in Table 1. The results on MNIST networks are very promising. We achieve state-of-the-art compression rates in both examples. We can furthermore show results for a light version of ResNet with 2.7M parameters to illustrate that our method does scale to modern architectures. We used more components (64)
here to cover the large regime of weights. However, for large networks such as VGG with 138M parameters the algorithm is too slow to get usable results. We propose a solution for this problem in Appendix C; however, we do not have any experimental results yet.
7 DISCUSSION AND FUTURE WORK
In this work we revived a simple and principled regularization method based on soft weight-sharing and applied it directly to the problem of model compression. On the one hand we showed that we can optimize the MDL complexity lower bound, while on the other hand we showed that our method works well in practice when being applied to different models. A short-coming of the method at the moment is its computational cost and the ease of implementation. For the first, we provide a proposal that will be tested in future work. The latter is an open question at the moment. Note that our method—since it is optimizing the lower bound directly—will most likely also work when applied to other storage formats, such as those proposed originally by Hinton & Van Camp (1993). In the future we would like to extend beyond Dirac posteriors as done in Graves (2011) by extending the weight sharing prior to more general priors. For example, from a compression point of view, we could learn to prune entire structures from the network by placing Bernoulli priors over structures such as convolutional filters or ResNet units. Furthermore, it could be interesting to train models from scratch or in a student-teacher setting.
ACKNOWLEDGEMENTS
We would like to thank Louis Smit, Christos Louizos, Thomas Kipf, Rianne van den Berg and Peter O’Connor for helpful discussions on the paper and the public code3.
This research has been supported by Google.
A REVIEW OF STATE-OF-THE-ART NEURAL NETWORK COMPRESSION
We apply the compression scheme proposed by Han et al. (2015b;a) that highly optimizes the storage utilized by the weights. First of all, the authors store the weights in regular compressed sparse-row (CSR) format. Instead of storing |W (l)| parameters with a bit length of (commonly) porig = 32 bit, CSR format stores three vectors (A, IR, IC).
• A stores all non-zero entries. It is thus of size |W (l)| 6=0 × porig, where |W (l)|6=0 is the number of non-zero entries in W (l).
• IR Is defined recursively: IR0 = 0, IRk =IRk−1+ (number of non-zero entries in the (k − 1)-th row of W (l)). It got K + 1 entries each of size porig.
• IC contains the column index in W (l) of each element of A. The size is hence, |W (l)|6=0 × porig.
An example shall illustrate the format, let
W (l) =  0 0 0 1 0 2 0 0 0 0 0 0 2 5 0 0 0 0 0 1  than
A = [1, 2, 2, 5, 1] IR = [0, 1, 2, 2, 4, 5]
IC = [3, 1, 0, 1, 3]
The compression rate achieved by applying the CSC format naively is
rp = |W (l)|
2|W (l)|6=0 + (K + 1) (9)
However, this result can be significantly improved by optimizing each of the three arrays.
A.1 STORING THE INDEX ARRAY IR
To optimize IR, note that the biggest number in IR is |W (l)|6=0. This number will be much smaller than 2porig . Thus one could try to find p ∈ Z+ such that |W (l)|6=0 < 2pprun . A codebook would not be necessary. Thus instead of storing (K + 1) values with porig, we store them with pprun depth.
A.2 STORING THE INDEX ARRAY IC
Instead of storing the indexes, we store the differences between indexes. Thus there is a smaller range of values being used. We further shrink the range of utilized values by filling A with zeros whenever the distance between two non-zero weights extends the span of 2pprun. Han et al. (2015a) propose p = 5 for fully connected layers and p = 8 for convolutional layers. An illustration of the process can is shown in Fig. 4. Furthermore, the indexes will be compressed Hoffman encoding.
A.3 STORING THE WEIGHT ARRAY A
In order to minimize the storage occupied by A. We quantize the values of A. Storing indexes in A and a consecutive codebook. Indexing can be improved further by again applying Huffman encoding.
B CONFIGURING THE HYPER-PRIORS
B.1 GAMMA DISTRIBUTION
The Gamma distribution is the conjugate prior for the precision of a univariate Gaussian distribution. It is defined for positive random variables λ > 0.
Γ(λ|α, β) = β α
Γ(α) λα−1e−βλ (10)
For our purposes it is best characterised by its mode λ∗ = α− 1 β and its variance varγ = α β2 . In our experiments we set the desired variance of the mixture components to 0.05. This corresponds to λ∗ = 1/(0.05)2 = 400. We show the effect of different choices for the variance of the Gamma distribution in Figure 5.
B.2 BETA DISTRIBUTION
The Beta distribution is the conjugate prior for the Bernoulli distribution, thus is often used to represent the probability for a binary event. It is defined for some random variable πj=0 ∈ [0, 1]
B(πj=0|α, β) = Γ(α+ β)
Γ(α)Γ(β) (πj=0)
α−1(1− πj=0)β−1 (11)
with α, β > 0. α and β can be interpreted as the effective number of observations prior to an experiment, of πj=0 = 1 and πj=0 = 0, respectively. In the literature, α + β is defined as the pseudo-count. The higher the pseudo-count the stronger the prior. In Figure 6, we show the Beta distribution at constant mode π∗j=0 = α− 1
α+ β − 2 = 0.9. Note, that, the beta distribution is a special
case of the Dirichlet distribution in a different problem setting it might be better to rely on this distribution to control all πj .
C SCALABILITY
Neural Networks are usually trained with a form of batch gradient decent (GD) algorithm. These methods fall into the umbrella of stochastic optimization (Robbins & Monro, 1951). Here the model parameters W are updated iteratively. At each iteration t, a set ofB data instances is used to compute a noisy approximation of the posterior derivative with respect to W given all data instances N .
∇W log p(W|D) = N
B B∑ n=1 ∇W log p(tn|xn,w) + I∑ i=1 ∇W log p(wi) (12)
This gradient approximation can subsequently be used in various update schemes such as simple GD.
For large models estimating the prior gradient can be an expensive operation. This is why we propose to apply similar measures for the gradient estimation of the prior as we did for the likelihood term. To do so, we sample K weights randomly. The noisy approximation of the posterior derivative is
now:
∇W log p(W|D) = N
B B∑ n=1 ∇W log p(tn|xn,w) + I K K∑ i=1 ∇W log p(wi) (13)
D FILTER VISUALISATION
In Figure D we show the pre-trained and compressed filters for the first and second layers of LeNet5-Caffe. For some of the feature maps from layer 2 seem to be redundant hence the almost empty columns. In Figure D we show the pre-trained and compressed filters for the first and second layers of LeNet-300-100.

This paper proposes to use an empirical Bayesian approach to learn the parameters of a neural network, and their priors.
A mixture model prior over the weights leads to a clustering effect in the weight posterior distributions (which are approximated with delta peaks).
This clustering effect can exploited for parameter quantisation and compression of the network parameters.
The authors show that this leads to compression rates and predictive accuracy comparable to related approaches.
Earlier work [Han et al. 2015] is based on a three-stage process of pruning small magnitude weights, clustering the remaining ones, and updating the cluster centres to optimise performance. The current work provides a more principled approach that does not have such an ad-hoc multi-stage structure, but a single iterative optimisation process.
A first experiment, described in section 6.1 shows that an empirical Bayes’ approach, without the use of hyper priors, already leads to a pronounced clustering effect and to setting many weights to zero.
In particular a compression rate of 64.2 is obtained on the LeNet300-100 model.
In section 6.1 the text refers to figure C, I suppose this should be figure 1.
Section 6.2 describes an experiment where hyper-priors are used, and the parameters of these distributions, as well as other hyper-parameters such as the learning rates, are being optimised using Spearmint (Snoek et al., 2012). Figure 2 shows the performance of the  different points in the hyper-parameter space that have been evaluated (each trained network gives an accuracy-compressionrate point in the graph). The text claims that best results lie on a line, this seems a little opportunistic interpretation given the limited data. Moreover, it would be useful to add a small discussion on whether such a linear relationship would be expected or not. Currently the results of this experiment lack interpretation.
Section 6.3 describes results obtained for both CNN models and compares results to the recent results of (Han et al., 2015) and (Guo et al., 2016).
Comparable results are obtained in terms of compression rate and accuracy.
The authors state that their current algorithm is too slow to be useful for larger models such as VGG-19, but they do briefly report some results obtained for this model (but do not compare to related work). It would be useful here to explain what slows the training down with respect to standard training without the weight clustering approach, and how the proposed algorithm scales in terms of the relevant quantities of the data and the model.
The contribution of this paper is mostly experimental, leveraging fairly standard ideas from empirical Bayesian learning to introduce weight clustering effects in CNN training.
This being said, it is an interesting result that such a relatively straightforward approach leads to results that are on par with state-of-the-art, but more ad-hoc, network compression techniques.
The paper could be improved by clearly describing the algorithm used for training, and how it scales to large networks and datasets.
Another point that would deserve further discussion is how the hyper-parameter search is performed ( not using test data I assume), and how the compared methods dealt with the search over hyper-parameters to determine the accuracy-compression tradeoff. Ideally, I think, methods should be evaluated across different points on this trade-off.